How to run Python Selenium with Flask on a VPS using PhantomJS






I love automation and web testing/bot making, the problem is I just can only run it on my local computer or make a GUI app. This week I tried to expand the use of it to the world-wide-web to get more audiences. Tired of doing "python __init__.py", then why not do "http://www.yourwebsite.com/script/" ???

I'm not going into detail how to do each part, you are required to have a general idea and a common sense how to debug and fix the problem. It took me prolly 2 weeks to figure out how to run the server correctly and fix all the basic bugs. Let's start.

1. You need to have a VPS, I'll recommend DigitalOcean.com for the cheap and free 2 months VPS. If you decide to register then please register under my referral link so I can get an earning from you and you can get 2 free months.

2.Install either LAMP stack or ngix on your VPS. I'm using LAMP stack so if you decidde to use ngix you have to figure out the problem on your own. (Note: I tried ngix, there is 1 problem that because the script usually takes longer to finish thna normal, so you always be timed-out.) The framework I use it Flask.

How to install LAMP stack on your VPS: https://www.digitalocean.com/community/tutorials/how-to-deploy-a-flask-application-on-an-ubuntu-vps
How to install Ngix/Unicorn on your VPS: https://realpython.com/blog/python/kickstarting-flask-on-ubuntu-setup-and-deployment/

Just follow all the setup and you will be ready to run python on a web. From now on I will just give instruction based on LAMP stack (Apache) only.

3. Make sure you are logged as root, now do "source venv/bin/activate", then "pip install selenium".

4. Now follow this to install PhantomJS, make sure you have a copy of the executable bin in your current "yourweb" folder. https://gist.github.com/julionc/7476620

5. PhantomJS needs a ghostdriver.log file in order to run, you have to create a customize log file because visitors cannot trigger the script to create that file in the system. So now in "yourweb", let create a blank "ghostdriver.log"

6. Supposed your web is in "var/www/yourweb/" then you have to set the group to www-data in order to let visitors execute the script. Just do "chown -R root:www-data /var/www/yourweb/".

Now up to setting permission, this problem took me a whole week to figure out how to do that:

7. Now you have "phantomjs" and "ghostdriver.log" inside "yourwebsite", let do chmod 750 for phantomjs and ghostdriver.log

8. Keep in mind, in order for visitors to run those files, all related folders have to be set to 750 also. Now do chmod 750 for "yourwebsite" AND "WWW"  <== I didn't chmod the "www" folder and It drove me crazy for couple days.

9. Now open /etc/sudoers, add these lines:
apache ALL=NOPASSWD: /var/www/yourweb/ghostdriver.log
apache ALL=NOPASSWD: /var/www/yourweb/phantomjs

I'm not quite understand the cause here, but if you don't do that you cannot run phantomjs and ghost driver correctly.

10. Now you are set and ready to go, make a simple Flaskapp and test it. Remember that your webdriver has to be customized to:

br = webdriver.PhantomJS(service_log_path='somewhere/ghostdriver.log', executable_path="somewhere/phantomjs")

Where somewhere is your absolute path to the folder that those files reside.

If you run into any problem, the log file will be saved in "/etc/log/apache2/error.log", just refer to the file.

Hope you could run selenium with flask on the website, and make an awesome app :D PM me if you need any help.

Comments

If you have any questions, please feel free to comment down here or reach me on my facebook page. I will get back to you as soon as I can! Thank you.

Archive

Contact Form

Send