Get Mechanize to handle cookies from an arbitrary POST (to log into a website programmatically)

I want to log into programmatically. My first idea was to use Mechanize to submit the login form:

alt text

However, it turns out that this isn't even a real form. Instead, when you click "Log in" some javascript grabs the values of the fields, creates a new form dynamically, and submits it.

"Log in" button HTML:

<button onclick="handleLogin(); return false;" class="btnBlue" id="myTMobile-login"><span>Log in</span></button>

The handleLogin() function:

function handleLogin() {
    if (ValidateMsisdnPassword()) { // client-side form validation logic
        var a = document.createElement("FORM"); = "form1";
        a.method = "POST";
        a.action = mytmoUrl; // defined elsewhere as
        var c = document.createElement("INPUT");
        c.type = "HIDDEN";
        c.value = document.getElementById("myTMobile-phone").value; // the value of the phone number input field = "txtMSISDN";
        var b = document.createElement("INPUT");
        b.type = "HIDDEN";
        b.value = document.getElementById("myTMobile-password").value; // the value of the password input field = "txtPassword";
        return true
    } else {
        return false

I could simulate this form submission by POSTing the form data to with Net::HTTP#post_form, but I don't know how to get the resultant cookie into Mechanize so I can continue to scrape the UI available when I'm logged in.

Any ideas?


You can use something like this to login and save the cookie so you won't have to do it again. Of course you will need to come up with your own logic to post it directly but this is how I use Mechanize's built in cookie_jar method to save cookies.

if !agent.cookie_jar.load('cookies.yml')
  page = agent.get('')

  form = page.forms.last = 'email'
  form.password = 'password'

  page = agent.submit(form)


I would avoid Net::HTTP; try with:

post(url, query={}, headers={})

directly from Mechanize class.

I often use the FireFox HttpFox extension to figure out what exactly is going on for these kind of problems.

Need Your Help

Single Sign On with Forms Authentication authentication forms-authentication single-sign-on

I am trying to set up Single sign on for 2 websites that reside on the same domain

Use Python to automate copying files to their corresponding target folders(shutil.copytree doesn't work as intended)

python file move shutil copytree

I wanted to automate the process of copying files (in their target folders) to their corresponding source folders (which has the same folder structure as the source) located in a different director...