URL Decoding during POST?

[Dani]

Hi,

I'm getting really frustrated with an issue that Ancient Dragon discovered is happening here. Anytime that the combination % 02 (without the space) is submitted via a textbox, it's recieved by PHP $_POST[] array just as a space. It's like URL decoding is happening on the fly. This happens with the message editor as well as with the search box. It happens regardless of whether JQuery's AJAX function is used to do the POST, or if it's just a regular HTML-based POST request.

<edit> Forgot to mention that we use the CodeIgniter PHP framework, if that helps.

[diafol]

I asume you're doing this sort of this in codeigniter:

$somedata = $this->input->post('some_data');

this usually uses the xss filter. I haven't looked at this myself, but it may be an untoward action of that?

I agree the post data is being urldecoded somewhere .

[veedeoo]

Hi,

You can also look at the form validation class in system/libraries/. Class definitions and functions are explained Here. Either extend the class as shown in this example, or set new rules

$this->form_validation->set_rules();.

I remember bumping into this problem two years back, but I can't even remember how and which files we have to tweaked. I will ask my brother Michael though to see if he ever remembers.

[diafol]

If you're using form validator:

$this->form_validation->set_rules('searchtext', 'Search', 'trim|required|min_length[3]|max_length[30]|xss_clean');

Check that the functions involved (trim etc) do not urldecode the data. BUT I think it may be the use of functions like xss_clean (which is rubbish anyway):

http://pastie.org/1682584

SHould give you an idea of what it does (if you're using it). It actually uses this line:

$str = rawurldecode($str);

[Dani]

I have CI's XSS Filtering disabled globally. I tried passing 'xss_clean' through the set_rules() function to enable it just there (as ardav mentioned) but that didn't work either.

At first I thought the problem was the code editor we use doing something funky, since it's very JavaScript heavy. Then I thought the problem might be related to the fact that we use JQuery AJAX to submit the form. But the problem seems to happen with our search box too, which is just a standard HTML form with a standard HTML textbox. At the very top of our search controller I'll do:

echo $this->input->post('query');
echo htmlspcialchars($this->input->post('query');
echo htmlentities($this->input->post('query');
echo urlencode($this->input->post('query');
echo rawurlencode($this->input->post('query');

Even better, I also tried

echo $_POST['query'];
echo htmlspecialchars($_POST['query']);
etc ...

and NOTHING seems to work. I have no idea who/what/where/when/why is manipulating my data. I looked at CodeIgniter's post() function in the Input class, and it just looks like it's taking directly from the $_POST[] superglobals array. But I tried fetching from $_POST directly and that didn't work either!

Grasping at straws, I also tried changing the HTML form to a multi-part form.

[Dani]

Oh, I just want to add, I can type %20 just fine, which, as far as I know, is what urlencode() turns a space into. Therefore, something funky is going on because not all percent-hex equivalents are being converted.

[diafol]

Odd. Sorry out of ideas at the mo. Good luck :(

[diafol]

The only place that I can find a ref to %20 is in Security.php and Upload.php both of which are to do with file names. Maybe? Will keep looking.

[cereal]

On CodeIgniter forum there is a thread about a similar problem (I think), but at the moment there aren't answers:

http://codeigniter.com/forums/viewthread/209004/

I'm wondering if this can be related to Utf8 library, this is in System/Core, check if removing @ from iconv() and mb_convert_encoding() will display an error. Good luck!

[Dani]

Hmm ... That does sound eerily similar to our problem. Let's try it ... %67.

[diafol]

What about at: core/common.php around line 512?

[Dani]

Wow ardav, I think you might be onto something with that function. For those of you out there, here is what ardav is referring to:

/**
 * Remove Invisible Characters
 *
 * This prevents sandwiching null characters
 * between ascii characters, like Java\0script.
 *
 * @access  public
 * @param   string
 * @return  string
 */
if ( ! function_exists('remove_invisible_characters'))
{
    function remove_invisible_characters($str, $url_encoded = TRUE)
    {

Let me investigate this one a bit further ...

[Dani]

%02

%02

Woot!!

ardav you are my hero right now!!!!! <3

[Ancient Dragon]

Testing

~~
printf("%02d", "something");
~~

[Dani]

AD, you need three or more tildes in a row. And the same number of opening ones as closing ones.