Click here to Skip to main content
15,038,345 members
Please Sign up or sign in to vote.
0.00/5 (No votes)
I need to send a post request to a form that is using 'https://' many times, as my script goes through a list of a lot of different words (the list is quite large). It works great only up to some point, and then it stops and I get the error:
events.js:291
      throw er;  // Unhandled 'error' event

Error: connect ETIMEDOUT some.ip.addr.es:443
   at tcp.connectWrap.afterConnect [as oncomplete] (net.js:1145:16)


'some.ip.addr.es' is not what shows in the error, I just changed the ip it showed.

I'm guessing I get the error because either I do the post request the wrong way in a FOR loop or the loop simply just does it too fast and too many times and it makes the connection time out. Because it works fine when it loops through a smaller list of words.

Here's my code:

var querystring = require('querystring');
var https = require('https');
const fs = require('fs');
var lineReader = require('line-reader');

// This is so that i can type the first_item manually
const readline = require('readline').createInterface({
    input: process.stdin,
    output: process.stdout
  });
  
  // Type my first_item here and pass it to my doWork function
  readline.question('first item: ', first_parameter => {
    doWork(first_parameter);
    readline.close();
  });

async function doWork(first_parameter) {
    if (first_parameter.length == 0) {
        console.log("You didn't specify the first item...")
    } else {
        // Read each line of my text file and pass it to post_php
        lineReader.eachLine('myList.txt', function(line, last) {
            post_php(line, first_parameter)
        })
    }
}

async function post_php(x, first_parameter) {
    process.env['NODE_TLS_REJECT_UNAUTHORIZED'] = '0';
    
    // My post data (body)
    var post_data = querystring.stringify({
    'first_item': first_parameter,
    'second_item': x,
});
    // The post request options (connection, php path etc)
    var post_options = {
        host: 'some.website.com',
        port: '443',
        path: '/directory/form.php',
        method: 'POST',
        headers: {
            'Content-Type': 'application/x-www-form-urlencoded',
            'Content-Length': Buffer.byteLength(post_data)
        }
    };
    
    // Connecting so that i can post to the form, there probably is a better way to do that, please let me know if so
    var post_req = https.request(post_options, function(res) {
        res.setEncoding('utf8');
        res.on('data', function (chunk) {

            // Tell me which word it is passing currently to the form
            console.log("Passing Parameter: " + x)
            });
        })

    post_req.write(post_data);
    post_req.end();
};


This code works good with a small list of words and throws no errors. With a bigger one, eventually I get hit with the ETIMEDOUT error and I'm assuming it's because either:

1. The loop is doing too many requests in too quick time and the server can't handle it (if so, how can I slow the loop down?)

OR

2. I'm not supposed to be sending https request to the php form this way and there is a better way to do that.

Please let me know if you can think of any possible fix for this. Thank you in advance.

What I have tried:

I tried to create a sleep() function and slow the loop down so that the https request isn't done too many times and too fast, but the sleep doesn't seem to affect anything and the loop still runs really fast.
Posted
Updated 23-Nov-20 18:43pm

1 solution

Start by making line-reader work with await:
JavaScript
import lineReader from 'line-reader';

function readLines(filename: string, processLine: (line: string) => Promise<void>): Promise<void> {
  return new Promise((resolve, reject) => {
    lineReader.eachLine(filename, (line, last, callback) => {
      if (!callback) throw new Error('panic');
      processLine(line)
        .then(() => last ? resolve() : callback())
        .catch(reject);
    });
  });
}
Create an async sleep function:
JavaScript
function delay(time, result) {
   return new Promise(function(resolve) {
       setTimeout(function() {
           resolve(result);
       }, time);
   });
}
Then combine the two to throttle your requests to the server:
JavaScript
async function doWork(first_parameter) {
    if (first_parameter.length == 0) {
        console.error("You didn't specify the first item...");
        return;
    }
    
    await readLines('mylist.txt', async (line) => {
        await post_php(line, first_parameter);
        await delay(100);
    });
}
You'll also need to update your post_php function to actually wait for the request to finish. As far as I can see, the low-level library doesn't support await, so you may need to use a different module - for example, node-fetch[^]
JavaScript
async function post_php(x, first_parameter) {
    process.env['NODE_TLS_REJECT_UNAUTHORIZED'] = '0';
    
    const params = new URLSearchParams();
    params.append('first_item', first_parameter);
    params.append('second_item', x);
    
    const post_options = {
        method: 'POST',
        body: params
    };
    
    const url = 'https://some.website.com/directory/form.php';
    const response = await fetch(url, post_options);
    if (!response.ok) {
        console.error(response.statusText);
    }
};
   
Comments
adamori 24-Nov-20 8:45am
   
This might work but now I'm getting another error.. unexpected ":" token.

Edit: fixed that by changing my extension from js to ts and running script with ts-node. However:

Now the process is drastically slower even without the sleep function and i dont get any errors. How can i speed things up to how they were before and only slow it down a little bit with the delay() function? I think your linereader function is what is slowing it down, but im not sure how to fix that.

If it cant be done, I just wanna Thank you anyways, your solution got me a lot closer to what im trying to achieve.

EDIT 2: Fixed it by not calling post_php function with await, now it's working the way I wanted it. Again, thank you so much for your solution.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)




CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900