Install python scapy on MAC OS X Sierra

Python ships by default with OSX and resides in the path /usr/bin/python. Having run into a lot of problems with installing and using scapy and other packages with the correct version of python, pip and homebrew, I decided to write steps that anyone can follow to setup a clean and working python environment in OSX without running it in a virtual environment.

Step 0: Check existing python version by the command `which python`. This should yield
$ /usr/bin/python

Step 1: Install homebrew
$ /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
(Reference: https://brew.sh/)

Step 2: Install python and pip using brew and pypcap using pip:
brew install python
pip install pypcap

Step 3: Setup your PATH variable correctly in ~/.bashrc to use the python installed by the above command in /usr/local/bin:
PATH=/usr/local:/usr/local/bin:$PATH
(add the above line at the end of your ~/.bashrc or ~/.zshrc, whichever shell you are using)

Step 4: Force updates to PATH variable by using source:
$ source ~/.bashrc

Step 5: At this point, if everything worked correctly, the output of `which python` and `which pip`should yield /usr/local/bin/python and /usr/local/bin/pip respectively

Step 6: Install scapy
$ pip install scapy

Step 7: Run scapy
If you get errors like "INFO: Can't import packageXYZ. ...", see step 8 below to fix dependencies.

Step 8: Fix Dependencies
i) pyx
(If using python 2.7): $ pip install pyx==0.12.1 -I --no-cache
(If using python 3): $ pip install pyx

ii) pycrypto
$ pip install pycrypto

iii) ecdsa lib
$ pip install ecdsa

Step 9: Test scapy again
19:55:35 Karans-MacBook-Pro ~: scapy
Welcome to Scapy (2.3.3)
>>>

The above shows a successful run of scapy with all dependencies correctly installed.

Accessing files on android via MTP on Linux

I recently purchased a Motorola MotoX from an AT&T store. Although it is a fabulous piece of hardware, the first trouble i encountered was getting access to files on my fedora 19. The internal memory was not being automatically mounted neither via MTP nor PTP. This was a bit frustrating.

However, the problem can be solved in 15-20 mins with a bunch of yum install commands. One needs to install gMTP and its dependencies which can be found here. Download the source code and extract it. Run ./configure, make and make install. If you encounter errors like 'Package somename was not found', you need to explicitly install it and re-run ./configure. I personally remember installing at least 5-7 dependencies before i got it working.

e.g. yum install fuse-devel libid3tag-devel libmtp-devel libmad-devel fuse fuse-libs libmtp simple-mtpfs glib2-devel

Once done, run gMTP and enable MTP mode on your android phone. You should now see the files listed in the gMTP window as shown below

Inmobi Hackathon 2014 : Experience and review

Last week, i attended Inmobi's hackathon organized at their Bangalore campus. It was dedicated to Aaron Swartz, a brilliant computer scientist who gave up his life while fighting surveillance and for a free and open internet world. Therefore, the hackathon was named as "FreedomHack". It all started the week before when one of my friends called up to discuss an idea which he thought was really worth implementing and wanted to know my opinion. Realizing that it was worth our time, i agreed and we formed a gang of 3 guys who registered to attend the 24 hour hackathon.

I reached at sharp 9am (yes, it was on Saturday and it is difficult for us engineers to get up early). However, i was excited about our idea and took all the required hardware with me in a back-pack. My team-mates also reached within 10-15 minutes. If i recall correctly, there were around 44-46 teams consisting of 3-4 members each. We were given our ID cards (labeled "Hackers"), were asked to give our notebook's serial number after which we were alloted a big space to work.

My ID Card

My Team(from left): Saurabh, Harshit & Me

We spent the first hour planning our hack on what we really wished to achieve and how we would divide the work among ourselves. Our app was called "Go Green". It had two use-cases:-

i) Given two points (a departing point and an arrival point, be it any city or location in the world), our application would mark those two points on a map and would display the best path to follow such that the carbon footprint is minimized.

ii) If one has already traveled in the past and his/her gmail account contains a copy of the ticket (the ticket maybe booked by famous travel websites like makemytrip.com/yatra.com), the application would fetch all travel details from one's email and show them within a single page. Then, if the user selects any of those journeys, the application would again plot the starting and end point of the journey and show the carbon footprint of the trip.

The basic idea of creating this application was two-fold:-

i) To make the users more aware of the increasing carbon-dioxide in the atmosphere and how they contribute to it.

ii) Help users in planning their future trips smartly by suggesting the best path to travel to minimize carbon footprint.

Furthermore, the application could generate revenue by displaying advertisements suitable to the mode of travel. So for example, if air-travel is involved, advertisements related to offers by airlines can be published. If one travels by road, then bus travel agencies can post their ads and so on.

Central point for free beverages

Inmobi offered us a feast throughout the event. They served breakfast, lunch on both days and dinner on the first day. Beverages and snacks could be consumed anytime by us throughout the event. In the evening, they also served burgers from McDonalds. Plus, we also got free Tshirts. There were some talks by invited persons on how to create better hacks and how to take our ideas to the next level. I felt that there should be no interruptions between the hackathon as it breaks concentration and rhythm. Talks are good at the start and end of the hackathon, but not quite in the middle.

                  

Workspace allocated to each team

Anyways, we completed 25% of our application by the first day's evening. The struggle started when we encountered issues during integration. We spent considerable time in optimizing code and how data from the lowest php based layer could be handed over to the upper layer which would ultimately be displayed in the User Interface. At around 2:30 am after several cups of tea/coffee, we started resolving our issues and by 6am, we had completed 90% of the application. It was the result of the good initial plan which we had laid down when we started. Demos started at 10:30 AM on sunday and the hackathon concluded by 2:30PM.

My teammate trying to get a nap (for hardly 5 mins)

It was a great experience. I learned a lot during a short time and saw some clever hacks by my peers. Thanks to everyone at Inmobi for organizing a wonderful event and giving us an opportunity to display our ideas. Lastly, thanks to my team-mates for staying up all night and helping each other in bug-fixing and even encouraging when one felt a little low.

Technorati Tags: inmobi, hackathon, experience, 2014, review

Battle of sorting algorithms (Which sorting algorithm is the best?)

I was recently going through MIT Opencourseware's Algorithms course by professor Erik Demaine and Charles Leiserson (fantastic teaching by the way!). After finishing up the lectures on sorting and worse case linear order statistics, an innocent question to ask is "Which sorting algorithm is best?". Mind you, the lectures are exhaustive and walk you through the running time of almost all discussed algorithms. But there are constraints and involvement of constants which hinder the practical applicability of these algorithms. In this post, i wish to delineate the algorithm selection mystery assuming you are sorting numbers. (You can extend the below mentioned analysis for floating points, strings and so on).

Algorithms based on comparing a pair of numbers and minimizing the number of comparisons have a tight lower bound of (n log n). This means that nlogn is the worst case runtime of the best comparison based sorting algorithm. And the best algorithm, in practice, is randomized quick sort.The advantage of randomized quick sort is that it avoids the worst case of quick sort (which might take upto O(n^2) time for selected inputs such as sorted or a reverse sorted array).

So, is randomized quick sort the answer to all problems?

As the lecture demonstrates, we can sort in linear time too. With algorithms such as counting sort and radix sort, it is quite possible to get a upper bound of O(n), where n is the number of elements to be sorted. These algorithms do not fall under the comparison based model.

So, are counting sort and radix sort the answer to all problems?

Not quite. Each of them is useful in their own way. For example, after going through the rigorous analysis presented in the lecture, i've come up with a criteria of when to use each of them:

1) Counting sort (Stable): Use it when

     i) You know the maximum number in the elements to be sorted.
     ii) All numbers are integers
     iii) You can afford to allocate two auxiliary storages of size equal to the number of elements.
     iv) The numbers themselves are not very large (perhaps less than 10,000). This limit can be increased if you can write a multi-threaded version of counting sort where you sort the numbers in parallel.

2) Radix sort (MSD implementations can be stable): Use it when

     i)  You know the range of numbers to be sorted
     ii) Elements to be sorted consists of only integers, fixed size strings or anything which can be converted to a fixed size integer.
     iii) The number of digits in each integer is less than log n where base of the log is the base of the number to be sorted. For example, if we desire to sort 10^6 integers which are all in base 10, then we'll check if log 10^6 base 10 is less than the number of digits in each integer. If yes, we'll go ahead with the sort. The logic of this check is explained in the lecture found here.

3) Randomized quick sort: Use it when

     i) When 1) and 2) cannot be applied.
     ii) You want to your algorithm to work well with cache.

Technorati Tags: MIT, algorithms, sorting, counting-sort, radix-sort, best-sorting-algorithm, quick-sort, randomized-quick-sort

Clarification on SIGKILL, SIGTERM, SIGINT, SIGQUIT, SIGSTP and SIGHUP

A few days ago, i landed upon unix signals that lead to process termination. I guess i was trying to remember the signals generated in linux when one presses Ctrl+Z and Ctrl+C. Memory did not serve me at that moment and i decided to look these up, one more time. I realized that having a consolidated book which explains these terms clearly is better than searching loads of webpages. I did the later since i had kept my unix os book away from my reach.To my disappointment, there was no single link that listed out all differences in an orderly fashion.

Hence, in this post, i wish to delineate these terms by consolidating my findings from stackoverflow, wikipedia and other unix internals websites. Here it goes:

SIGKILL: Terminates a process immediately. This signal cannot be handled (caught), ignored or blocked. (The "kill -9" command in linux generates the same signal).

SIGTERM: Terminates a process immediately. However, this signal can be handled, ignored or caught in code. If the signal is not caught by a process, the process is killed. Also, this is used for graceful termination of a process. (The "kill" command in linux if specified without any signal number like -9, will send SIGTERM)

SIGINT: Interrupts a process. (The default action is to terminate gracefully). This too, like, SIGTERM can be handled, ignored or caught. The difference between SIGINT and SIGTERM is that the former can be sent from a terminal as input characters. This is the signal generated when a user presses Ctrl+C. (Sidenote: Ctrl+C denotes EOT(End of Transmission) for (say) a network stream)

SIGQUIT: Terminates a process. This is different from both SIGKILL and SIGTERM in the sense that it generates a core dump of the process and also cleans up resources held up by a process. Like SIGINT, this can also be sent from the terminal as input characters. It can be handled, ignored or caught in code. This is the signal generated when a user presses Ctrl+\.

SIGSTP: Suspends a process. This too, can be handled, ignored or blocked. Since it does not terminate the process, the process can be resumed by sending a SIGCONT signal. This signal can be generated by pressing Ctrl+Z. (Sidenote: Ctrl+Z stands for substitute character which indicates End-of-File in DOS)

SIGHUP: (From Wikipedia): Hangs up a process when the controlling terminal is disconnected. This especially relates to modem/dial in connections. A process has to explicitly handle this signal for it to work. A good use is to "poke" a process and letting the process (as defined by the programmer) decide what to do with the signal is described here. Hence, SIGHUP can be handled, ignored or caught. This is the signal generated when a user presses Ctrl+D.

Technorati Tags: unix, signals, Unix-signals, SIGINT, SIGTERM, SIGKILL SIGQUIT, SIGSTP, SIGHUP

Designing a good security policy for your websites

Recently, i went through web server security and security analysis of forums based on phpBB. Although i do agree that humans are the weakest link in the security chain, there are a few measures a web developer can take to prevent malicious users from exploiting their website.

1) The login page (or any other page that requires authentication) should use HTTPS. The best approach is to use HTTPS throughout your website.

2) Set secure cookies and check them on each request from the user. This goes hand in hand with step 1.

3) Limit the maximum number of login attempts (say 5). Also, use techniques like exponential delay at each failed login attempt.

4) Deploy captcha verification on pages that require authentication (probably couple the captcha with the login page). Captcha should be account based rather than IP based (doing so prevents DDOS attacks since step 3 should stop login attempts after a few incorrect tries regardless of the IP used to log into the account)

5) After the maximum number of invalid login attempts have been reached, a web developer can go with two approaches.
     i) Deactivate the account and send password reset link at the user's registered email.
                                                                      OR
    ii)  Throw up a security question (entered earlier by the user) and follow the deactivation step if this too fails.

6) Set a password complexity policy (make digits, mixed alphabets and special characters mandatory along with a minimum password length)

7) Change the session ID on each request (for websites that require extra-security). Also, make sure that the session ID or any other session information is not a part of the query string (this will not be much of a problem if HTTPS is used throughout the website)

8) Force logout after a fixed period of inactivity.

9) Make extra authentication checks for administrator logins.

10) Never trust user input. Always validate it.

11) Maintain a blacklist (or whitelist) of users (IP based or whatever suits you best) to block malicious users who try to attack your website. It's not recommended to block them forever.

12) Understand and write your robots.txt with diligence.

13) Use vulnerability scanners to scan your website for security loopholes. Some good ones are jsky, acunetix and w3af. Fix the issues these scanners list and re-scan to confirm from various other scanners.

 14) Take regular backups of your website. In case of a security incident, the migration to another webserver or to restore the site back will be seamless with proper backups.

15) Monitor your website traffic and statistics on a regular basis. Watch out for unusual traffic (depending on IP, location, webpage requested and so on). Use google's webmaster's tool and google analytics to aid you in the development and monitoring process.

These are just my thoughts. Feedback or any other addition to the above listed policy is appreciated.

Technorati Tags: security, Policy, Web security, phpBB

Mobremote: An insider's view

A usual morning at college with an ongoing data structures class suddenly hit us (Me and Nitesh) with a new software idea. Remote Control via text messages!

A lot has already been achieved in Remote Control, now that mobile devices are faster, slimmer and easy to use. But we had a different idea in our mind. To create something that eliminates the cost factor of a fancy cellphone. We started searching and we came across loads of different software, but we found some things common in almost all of them:

1. Either the user requires a second cellphone attached to his computer and controls the computer via another cellphone with him.

2. The cellphone as well as the computer have a fast internet connection and the client running on the computer talks with the cellphone via a protocol delivering the computer's current state on the cellphone's screen.

3. The cellphone updates a part of a common web server/user profiles (3rd party over which user has very little control) and the client would read the profile. This mostly is facilitated via a sms gateway (generally owned by a 3rd party website) which is most cases demands premium sms charges or subscription to their service.

In all the above mentioned, there are charges involved that the user needs to pay in order to control his computer remotely, either overhead of a second cellphone, bandwidth charges or lastly, premium sms or subscription charges.

The solution of these problems became the central idea behind our project and we named it MOBREMOTE. (short for Mobile Remote)


Mobremote provides the user with ability to send text messages containing the desired command to execute on his computer. There is absolutely NO need of an internet connection on the user's cellphone. The only requirement is that of an existing gmail account and the client computer only requires a minimal internet connection (even dialups would do!) which is generally cheaper than the cost of an additional cellphone and mostly better than subscription charges.

Advantages:
1. NO premium charges of sending the text message
2. NO internet connectivity required on the cellphone (saving extra bandwidth charges)
3. NO monthly subscription charges
4. Full Control over what can be remotely done with the cellphone (with different layers of security at each level)


So how do implement our idea? It's plain and simple. We came across a platform known as txtWeb which runs in India, United States Of America and Canada. The platform is run by Intuit, based in United States itself. Their primary aim is to bring the power of internet to user's cellphone for no extra charges except the cost of a standard text message. txtWeb has dedicated numbers assigned for each of these countries.

The path that we followed in implementing Mobremote was:

User's cellphone sends message at txtweb's number->> Process message and check for correctness->> Forward the message's content to user's specified gmail account->>Create a separate application to run on user's computer->>The application would periodically check for incoming commands in user's gmail account->> If the received message from user's gmail inbox passes all security checks, execute the specified command in the message.

This was the ultimate solution for us to go ahead and bring convert our idea into reality. So we started building a text-message handling application. We hosted the application on Google AppEngine for reliability, security as well as diagnostic reasons. Next we map a keyword on txtWeb's website that would point to our application hosted on AppEngine. By doing so, we ensure that when the user sends a message with our keyword and his details to txtWeb, txtWeb would in turn run our application with the message as a parameter and return the result back to the user (or forward the message to user's gmail account, sending back a success message to user's cellphone). We designed a format in which the user is required to specify his details in the text message.

Mobremote's current format is:
@mobremote -k (security key) -e (gmailid@gmail.com) -s command -c (command to execute)

Security key (5 digits entered in the Mobremote software installed on the user's computer) is implemented as a part of Mobremote’s security in order to protect the user’s computer from unwanted/fake commands from any other person other than the owner of the computer himself. The user is required to fill in the details specified in braces and send the text message to one of the txtWeb's numbers (9243342000 or 9900173324 for India, and 898932 for US and Canada).

The command is specified as digits ending with a $ sign. Mobremote currently supports 9 most basic commands the user may need when he/she is not physically near their computer.
The number to command mapping is as follows:

0:Stop listening to incoming commands from cellphone

1:Lock computer

2:Sleep Computer

3:Hibernate computer

4:Log off user

5:Restart computer

6:Shutdown computer

7:Force shutdown computer

8:End Unresponsive program


EXAMPLE: If your gmail id is dev@gmail.com with the configured security key as 12345 and you want to restart your computer, you will send to the appropriate txtWeb number: @mobremote -k 12345 -e dev@gmail.com -s command -c 5$

We offer several help options to our users via text messages like:

1. To get information on mobremote, send “@mobremote” (without quotes) to the above txtWeb number which applies to you according to your location.
2. To get the format of sending commands, send “@mobremote format”
3. To get a list of supported commands, send “@mobremote help”
4. To view an example, send “@mobremote example”


So, uptil now, our text message has travelled from user's cellphone to his/her gmail account. As stated above, we create another application that would run on the user's machine and regularly check for any incoming emails in the format of Mobremote. So we started writing code to create a good user interface along with added security to extend what the user can accomplish by using Mobremote. We wrote code in C++ utilizing Qt framework to provide a user friendly GUI. The user is required to fill in his details in the Mobremote software which includes his gmail account details, mobile number and security key and set the software in Running mode in order to listen to incoming commands from user's cellphone.

The Gmail ID and security key is required to be sent in the text message from user’s cellphone, so the user needs to remember these details entered on his/her computer. Mobremote executes commands only when it verifies user’s identity via his/her gmail account ID and security key.

The software installed on the end user's machine uses PHP with IMAP+SSL to securely fetch email messages containing commands sent from user's cellphone. To provide a scripting environment, we require installation of Apache.

We bought our own domain www.mobremote.com and created a website for our project.

After 6 months of coding, testing, debugging, finally we packed everything into a installer and it was ready to be distributed. We uploaded both 32 and 64 bit installer for Windows XP/Vista/7 along with Wampserver (contains Apache for Mobremote to make use of) in the download section of our website www.mobremote.com/download

Developing Mobremote was truly a learning and fun-filled experience. Of course, there were times when we were so frustrated that we wanted to smash our computer itself, but in the end it worked out well and we were good to go. Mobremote is written in around 5 computer languages all working together to provide remote control facility to the user. Ahead of time, we plan to add more features to Mobremote, including starting/stopping applications, automated downloads, facebook/twitter status updates and the list goes on!

Mobremote was developed with an open mind so it's free to use and re-distribute. However, we do not share our source code with anyone. Developers who are willing to contribute to our project can visit For Devs for more details.

Technorati Tags: mobremote, remote, sms, mobile, cellphone, software, free, txtweb, remote-control