Topic: We have some scraper bot makers here right?

Anonymous A started this discussion 15 years ago #5,206

Ive made a bot, that logs into a proboards forum. What i want it to do is get the whole html (which it does), but i want a way using regex to split the thread id and title of the thread which is in the setWindowStatus brackets and store them in an array or something so i can do like

mylinks[0] = Threadid :: Thread title

<img src="http://s2.images.proboards.com/xx.gif" alt=" " border="0"/></font></td><td class="windowbg" bgcolor="FFFFFF" width="48%" style="cursor:pointer;" onClick="if(!pb_bubble)location.href='/index.cgi?board=general&action=display&thread=13483';" onMouseOver="mouseOverHighlightCell(this);setWindowStatus('Lurkers, show your face ');return true;

Anonymous B joined in and replied with this 15 years ago, 16 minutes later[^] [v] #91,841

What is the bot written in? Can you give an example of html you are grabbing -- Maybe a link to a page from the forum or a page on a similar forum?

Katerina Shcherbatskaya joined in and replied with this 15 years ago, 32 minutes later, 49 minutes after the original post[^] [v] #91,869

/thread=([0-9]+).*?setWindowStatus\('(.*?)'\)/

Anonymous D joined in and replied with this 15 years ago, 11 minutes later, 1 hour after the original post[^] [v] #91,878

its in php. I think all proboards forums are the same. That one is password protected though, so cant show you a page from there

Anonymous B replied with this 15 years ago, 30 minutes later, 1 hour after the original post[^] [v] #91,912

@91,869 (Katerina Shcherbatskaya)
Yeah, this

protip !!pMmAiEwOm joined in and replied with this 15 years ago, 56 minutes later, 2 hours after the original post[^] [v] #91,983

@91,869 (Katerina Shcherbatskaya)
Thatâ€™s almost too specific.

This is possibly too loose: /[&|?]thread=[0-9]+/i

This is a more programmatic match that can be used for other tasks (untested)â€¦



function collect_all_href_thread_ids($full_HTML_as_string_input) {

    // output is an array of href strings

    $href_thread_ids = $query_args = array(); 

    

    // the pattern in onClick attributes for images

    // example: /index.cgi?board=general&action=display&thread=13483

    $full_href_pattern = /location.href=["|'](\w+)['|"]/g;

    

    // full page match for all hrefs that collects the query strings

    preg_match_all($full_href_pattern, $full_HTML_as_string_input, $all_matching_hrefs, PREG_PATTERN_ORDER);

    

    // skip the first, as itâ€™s the full-pattern matches

    next($all_matching_hrefs);

    

    // iterate through each match

    while(list($key, $query_string) = each($all_matching_hrefs)){

        // get all key:value pairs

        $each_query_arg = explode($query_string, '&');

    

        // map the strings to machine-ready, string-keyed arrays

        while(list($k, $v) = each($query_args)){

            $my_args = explode($v, '=');

            $query_args[$my_args[0]] = $my_args[1];

        }

    

        // append the thread ID to the output

        $href_thread_ids[] = $query_args['thread'];

    }



    return $href_thread_ids;

}

Katerina Shcherbatskaya replied with this 15 years ago, 1 day later, 1 day after the original post[^] [v] #93,057

@previous (protip !!pMmAiEwOm)
It needs to capture the thread title too, though.

Minichan