Python HTTP Redirect requests forbidden -

-

i'm trying scrape website url redirected, programmatically trying gives me 403 error code (forbidden). can place url in browser , browser follow url though...

to show simple example i'm trying go : http://en.wikipedia.org/w/index.php?title=mike_tyson

i've tried urllib2 , mechanize both not work. new web programming , wondering whether there other tricks need in order follow redirect!

thanks!

edit

okay, messed. looking alternative methods because trying scrape mp3. managing succesfully downloading mp3 mangled.

turns out somehow related me downloading on windows or current python version. tested code on ubuntu distro , mp3 file downloaded fine....

so used simple urllib2.openurl , worked perfect!

i wonder why downloading on windows mangled mp3?

try changing mechanize flag not respect robots.txt. also, consider changing user-agent http header:

>>> import mechanize >>> br = mechanize.browser() >>> br.set_handle_robots(false) >>> br.addheaders = [('user-agent', 'mozilla/4.0 (compatible; msie 6.0; windows nt 5.1; sv1)')]

web servers treat running ms internet explorer 6, rather bot. if restrict robots.txt, bot continue work until blocked.

>>> br.open('http://en.wikipedia.org/w/index.php?title=mike_tyson') <response_seek_wrapper @ 0x... wrapped object = <closeable_response @ 0x... fp = <socket._fileobject object @ 0x...>>> #doctest: +ellipsis

Comments

Post a Comment

Popular posts from this blog

ASP.NET/SQL find the element ID and update database -

-
basically i've got update query runs when click submit button. i have asp:repeater in page layer inside this <asp:repeater id="dgbookings" runat="server" onitemdatabound="itemdb"> <itemtemplate> <div class="bookingscontent"> <div class="bookingdetails" id="<%# databinder.eval(container.dataitem, "booking_ref") %>"> <p class="infotext"><%# databinder.eval(container.dataitem, "adults") %> <asp:linkbutton onclick="editdetails" text="edit..." runat="server"></asp:linkbutton></p> <p class="infotext"><asp:linkbutton onclick="submitdetails" text="submit..." runat="server"></asp:linkbutton></p> </div> </div> </itemtemplate> <asp:repeater> and outputs ...
Read more

jquery - appear modal windows bottom -

-
i have modal windows in application, work fine <div id="modal" class="modal hide fade" tabindex="-1" role="dialog" aria-labelledby="mymodallabel" aria-hidden="true"> but when modal appears, appear top of windows, , want appear bottom the application joomla 3.0! , using template of meet gavern any idea? i found solution i modified de css in bootstrap .modal.fade { top:-25%; -webkit-transition:opacity .3s linear, bottom .3s ease-out; -moz-transition:opacity .3s linear, bottom .3s ease-out; -ms-transition:opacity .3s linear, bottom .3s ease-out; -o-transition:opacity .3s linear, bottom .3s ease-out; transition:opacity .3s linear, bottom .3s ease-out; }
Read more

c++ - Compiling static TagLib 1.6.3 libraries for Windows -

-
i having super hard time compiling , using taglib 1.6.3 in qt project. i've tried can think of. taglib claims supported through cmake i'm not having luck. furthermore, i'm confused kinds of files need qt libs! i've built *.a files, *.lib, , *.dll. understand far... believe since i'm working in windows *.lib want. no matter do, end "undefined references" taglib functions try use when try compile qt project. have tried mingw32, msys, visual studio 2008, , cross-compiling windows on linux. turning nothing. what makes less sense me if compile same taglib source qt on mac (g++ think?) works fine! somewhere in windows compilation procedures have going wrong. have been smacking face on desk 30 (on , off) hours trying figure out. since qt uses mingw must compile taglib same compiler? if compile *.lib's visual studio not compatible? are *.a libraries usable in windows? (assuming mingw) i'm still trying handle on c++ stuff, after reading coun...
Read more