Friday, December 30, 2016

Convert String to Base64 in Python

Code to convert string (from utf-8) to base64 encoding and base64 back to utf-8:

import base64

si = input('\nenter text:\n')
print('\nyou entered\n' + si)

b64 = base64.b64encode(bytes(si, 'utf-8'))
print('\nbase64 is\n' + str(b64))

b64_bs = bytearray(base64.b64encode(bytes(si, 'utf-8')))
print('\nbase64 byte stream is\n' + str(list(b64_bs)))

so = base64.b64decode(b64).decode('utf-8', 'ignore')
print('\nstring is\n' + so)

Convert String to Binary in Python

Code to convert string to binary and binary to string:

def string2bits(s=''):
    return [bin(ord(x))[2:].zfill(8) for x in s]

def bits2string(b=None):
    return ''.join([chr(int(x, 2)) for x in b])

si = input('\nenter text:\n')
b = string2bits(si)
so = bits2string(b)

print('\nyou entered\n' + si)

print('\nbinary is\n' + str(b))

print('\nstring is\n' + so)

TypeError: Can't convert 'bytes' object to str implicitly


Traceback (most recent call last):
  File "", line 10, in <module>
    print('\nstring is\n' + so)
TypeError: Can't convert 'bytes' object to str implicitly

Problem Code:

so = base64.b64decode(b64)
print('\nstring is\n' + so)


so = base64.b64decode(b64).decode('utf-8', 'ignore')
print('\nstring is\n' + so)

AttributeError: module 'json' has no attribute 'dump'


Today I was trying to create a json file from Python and then I got this error: "AttributeError: module 'json' has no attribute 'dump'" and I found my solution here:


So the problem was I named my file as that was causing problems with json module so renaming it solved the issue.


import json
with open("dump.json", "w") as outfile:
    json.dump({'number':6650, 'strings':'lorem ipsum', 'x':'x', 'y':'y'}, outfile, sort_keys = True, indent=4, ensure_ascii=False)


Make file read-only in Python

If you want to make file read-only:

import os
from stat import S_IREAD

os.chmod('path/to/file.txt', S_IREAD)

If you want to make file writeable:

import os
from stat import S_IWRITE

os.chmod('path/to/file.txt', S_IWRITE)

Thursday, December 29, 2016

Create PDF from Python

I was googling for a python library that I can use to write strings to a PDF file but mostly I found out programs to copy data from one PDF and write it to another PDF, another programs and solutions I found were for editing existing PDF, it was not what I was looking for.

I wanted to create a PDF from scratch.

After hours of searching finally I found my solution when I arrived on this page by David Fischer and it just uses one module called reportlab.

Code is share via this gist on my GitHub.

TypeError: write() argument must be str, not bytes

Recently I encountered this error: "TypeError: write() argument must be str, not bytes" when trying to write a PDF file from anaconda 3. The solution was to write stream in binary.


Traceback (most recent call last):
  File "", line 31, in <module>
TypeError: write() argument must be str, not bytes

Problem code:

# Write the PDF to a file
with open('test.pdf', 'w') as fd:


# Write the PDF to a file
with open('test.pdf', 'wb') as fd:

Wednesday, December 28, 2016

Extract all internal and external links from a URL

How it works?
  • This is a Python script which takes complete URLs provided as command line arguments.
  • It then parses a URL in HTML using BeautifulSoup.
  • From the parsed webpage all the anchor hyper-references are extracted and later simple processing is done to sort them in two bins: internal and external.
  • If link contains http or https and the URL is a part of link then it is sorted as internal link otherwise it is external link.
  • All links starting with / or // are internal links.
  • Other tags like javascript, mail and telephone links are ignored.
  • All internal page jumps starting with # are ignored.
  • This link does not provide unique list of internal/external links so if same link is present it will be counted multiple times.
  • It will treat Top-level domain and subdomains as different URL hence it will be counted as external i.e. if you are querying for then links with and both will be treated as different domain thus will be counted to external links.

Tuesday, December 27, 2016

Solve charmap codec error in Python


'charmap' codec can't encode character '\u2013' in position 112: character maps to <undefined>

Problem code:

        for d in soup.find_all(href=re.compile(url)):

It is not an error from Python but it is because of Windows, in my case there was one character which was not getting encoded properly in command prompt (console) so I had to change the encoding to UTF-8. Solution is pretty simple just type following command.


J:\>chcp 65001
Active code page: 65001

CHCP changes the active console code page. 65000 code page is encoded as UTF-7 and 65001 code page is encoded as UTF-8.

Friday, December 09, 2016

YouTube Rewind (2010 - 2016)


YouTube Rewind is a video series produced and created by YouTube and Portal A Interactive. These videos are an overview and recap of each year's viral videos, events, memes and music. Each year, the number of YouTube celebrities featured in the video, as well as the presentation of the series have increased.


YouTube Rewind 2010: Year in Review

Saturday, October 01, 2016

Run a server from Node.js

Step 1: Download and install Node.js from

Step 2: Launch node from command prompt and verify it has installed correctly

Step 3: Create a new file server.js as follows:

var http = require('http');
var fs = require('fs');
var url = require('url');

http.createServer( function (request, response)
   var pathname = url.parse(request.url).pathname;
   fs.readFile(pathname.substr(1), function (err, data)
      if (err)
         response.writeHead(404, {'Content-Type': 'text/html'});

Create JSON from HTML in Perl

Assumption: Your HTML file is not pretty-encoded i.e. all content is contained in a single line.



use strict;
use warnings;
use utf8;
use JSON;
use autodie;

open my $fh1, '<files.html' or die "Cannot open files.html: $!\n";

my $html = <$fh1>;

$html =~ s/<html><head><title>JSON Viewer<\/title><\/head><body><table border="1"><thead><tr><th width="300">Keys<\/th><th width="500">Values<\/th><\/tr><\/thead><tbody>//g;
$html =~ s/<\/tbody><\/table><\/body><\/html>//g;

my @rows = split("<\/tr>", $html);

my %data;

foreach my $row (@rows)
    my @cols = split("<\/td>", $row);
    my $key;
    foreach my $col (@cols)
        if (not $col =~ /<br\/>/)
            $col =~ s/<tr><td>//g;
            $key = $col;
            my @values = split("<br\/>", $col);
            foreach my $value (@values)
                $value =~ s/<td>//g;
                push @{ $data{$key} }, $value;

close($fh1) or die "Cannot close files.html: $!\n";

my $json = encode_json \%data;
open my $fh2,">files.json" or die "open failed <output: files.json>: $!\n";
print $fh2 $json or die "print failed <output: files.json>: $!\n";
close $fh2 or die "close failed <output: files.json>: $!\n";

Create HTML from JSON in Perl


use strict;
use warnings;
use utf8;
use JSON;
use autodie;
use HTML::TreeBuilder;

my $string;
    local $/;
    open my $fh1,"<files.json" or die "open failed <files.json>: $!\n";
    $string = <$fh1>;
    close $fh1 or die "close failed <files.json>: $!\n";

# Parse JSON in PERL
my $data = decode_json($string);

# Initialize HTML
my $html = "<html><head><title>JSON Viewer</title></head><body><table border=\"1\"><thead><tr><th width=\"300\">Keys</th><th width=\"500\">Values</th></tr></thead><tbody>";

# Update HTML
foreach my $key (keys(%$data))
    $html = $html."<tr><td>".$key."</td><td>";
    my $values = $data->{$key};
    foreach my $value (@{$values})
        $html = $html.$value."<br/>";
    $html = $html."</td></tr>";
# Generate HTML
$html = $html."</tbody></table></body></html>";
my $h = HTML::TreeBuilder->new_from_content($html);
open my $fh2,">files.html" or die "open failed <files.html>: $!\n";
print $fh2 $h->as_HTML('',"\t") or die "print failed <files.html>: $!\n";
close $fh2 or die "close failed <files.html>: $!\n";

Monday, September 12, 2016

Send email from Perl in Windows 10

In this program I am using Gmail to send an email from Strawberry Perl on my Windows 10 system.

You can install Perl from here:

I installed latest Strawberry Perl (64bit) msi and it works fine on Windows 10 Pro 1607.

Next install few modules that are required to send email.

Open start menu and type CPAN Client that was installed on your PC by Strawberry Perl.

Open CPAN Client and type following commands to install:
install Email::Send
install Email::Send::Gmail
install Email::Simple::Creator

As we are using gmail for sending email you need to authenticate your account against gmail's smtp server with your gmail credentials and make sure you have enables POP and IMAP email forwarding on from gmail settings.

Change your credentials in following program:

Sunday, August 28, 2016

Top 9 Android Smartphones For 15000 Rupees

1. Xiaomi Mi Max

Camera: 16/5 MP
Display: 6.44"
Storage: 32 GB
Released: May 2016

2. Asus Zenfone 2

Camera: 13/5 MP
Display: 5.5"
Storage: 16 GB
Released: March 2015

3. Vivo V 3

Camera: 13/8 MP
Display: 5"
Storage: 32 GB
Released: April 2016

Fix update/upgrade errors in Kali Linux [Rolling Release 2016.1]

Last year I published an article on this blog about fixing update and upgrade issues in Kali Linux and it helped many users and now again after a year this issue has occurred which can not be solved by previous trick hence posting a new article here. Check previous article:

This trick is better than previous one. All you need to do is check latest repository links and update it in your sources list. You can go to official Kali Linux website documents and check for the current repository.

Then open /etc/apt/sources.list in your favorite editor and overwrite the file with latest repositories links you found from official website.

For now you can gedit /etc/apt/sources.list and change it to as this is for latest version Kali Linux 2016.1 Rolling Release:

deb kali-rolling main contrib non-free
# For source package access, uncomment the following line
# deb-src kali-rolling main contrib non-free

That's it now you can run apt-get clean && apt-get autoremove && apt-get update && apt-get upgrade && apt-get dist-upgrade and it should work fine.

Android 7.0 Nougat list of supported SOC

Supported processors are as below:
  • Qualcomm Snapdragon
  1. 415
  2. 430
  3. 435
  4. 615
  5. 616
  6. 617
  7. 625
  8. 650
  9. 652
  10. 805
  11. 808
  12. 810
  13. 820
  14. 821
  • Samsung Exynos ARMv8
  1. 5433
  2. 7420
  3. 7870
  4. 8890
  • Huawei Kirin 950, 955
  • MediaTek MT with Mali
  1. 6732
  2. 6722
  3. 6755
  4. 6797
  • MediaTek MT with PowerVR
  1. 6595M
  2. 6595T
  3. 6595M
  4. 6795
  5. 8135
  6. 8173
  7. 8176
  8. 679x
  • Intel Atom Z
  1. 3460
  2. 3480
  3. 3530
  4. 3560
  5. 3570
  6. 3580
  • Rockchip RK 3368
  • Nvidia Tegra K1, X1
  • Nvidia GeForce
  1. 600
  2. 700
  3. 800
  4. 900
  5. 1000

Unsupported processors are as below:
  • Qualcomm Snapdragon
  1. 200
  2. 208
  3. 210
  4. 212
  5. 400
  6. 410
  7. 412
  8. 600
  9. 800
  10. 801
  • Samsung Exynos
  1. 5250
  2. 5260
  3. 5410
  4. 5420
  5. 5422
  6. 5430
  7. 5800
  8. 7580
  • Mediatek MT 6735, 6753
  • Huawei Kirin
  1. 920
  2. 925
  3. 930
  4. 935
Learn more about Android 7.0 Nougat here:

Saturday, July 16, 2016

Learn C and C++

C and C++:

The major difference between C and C++ is that C is a procedural programming language and does not support classes and objects, while C++ is a combination of both procedural and object oriented programming language; therefore C++ can be called a hybrid language.


C was developed by Dennis Ritchie between 1969 and 1973 at AT&T Bell Labs. When compared to C++, C is a subset of C++. C supports procedural programming paradigm for code development. In C (because it is a procedural programming language), data and functions are separate and free entities.

Full (MEAN) Stack Developer Tutorials

Full Stack Development:

The term full-stack means developers who are comfortable working with both back-end and front-end technologies. To be more specific, it means that the developer can work with databases, PHP, HTML, CSS, JavaScript and everything in between, also, venturing as far as converting Photoshop designs to front-end code. (Ref.:

MEAN Stack:

MEAN is a framework for an easy starting point with MongoDB, Node.js, Express, and AngularJS based applications. It is designed to give you a quick and organized way to start developing MEAN based web apps with useful modules like Mongoose and Passport pre-bundled and configured. We mainly try to take care of the connection points between existing popular frameworks and solve common integration problems. (Ref.:

Online Certification Courses (MOOCs) and Tutorials:

Fix installation error in Anaconda

What is Anaconda?

Anaconda is the leading open data science platform powered by Python. The open source version of Anaconda is a high performance distribution of Python and R and includes over 100 of the most popular Python, R and Scala packages for data science.

The error reads as below:

Traceback (most recent call last):
File "C:\Anaconda2\Lib\", line 164, in <module> main()
File "C:\Anaconda2\Lib\", line 150, in main
File "C:\Anaconda2\Lib\", line 94, in mk_menus
  err("Traceback:\n%s\n" % traceback.format_exc(20))
IOError: [Errno 9] Bad file descriptor

Active Directory login in GitBlit

What is Git?

Git is a version control system that is used for software development and other version control tasks. As a distributed revision control system it is aimed at speed, data integrity, and support for distributed, non-linear workflows. Git was created by Linus Torvalds in 2005 for the development of the Linux kernel, with other kernel developers contributing to its initial development.

What is GitBlit?

Gitblit is an open-source, pure Java stack for managing, viewing, and serving Git repositories. It's designed primarily as a tool for small workgroups who want to host centralized repositories.

Read more about GitBlit:

Thursday, April 28, 2016

Host shiny server in Ubuntu 14.04 LTS

I have installed shiny server on Ubuntu 14.04 LTS running under a VM. This post is a collection of commands that I had used for setup. Note that if you want to use shiny server on LAN then use Bridged Network Adapter.

On VM you can access the shiny server via localhost:3838 and on remote machines, you can use IP-address:3838 to open it. You can host your shiny app in the /srv/shiny-server/ directory. You can go to /etc/shiny-server/shiny-server.conf to change default configurations like you can change port with listen <port number>; in server {} module.

$ sudo sh -c 'echo "deb trusty/" >> /etc/apt/sources.list'

$ sudo gpg --keyserver --recv-key E084DAB9

$ sudo gpg -a --export E084DAB9 | sudo apt-key add -

$ sudo apt-get update

$ sudo apt-get -y install r-base

$ sudo su - -c "R -e \"install.packages(c('shiny','shinyjs','shinydashboard','shinythemes','devtools','rmarkdown'), repos = '')\""

$ sudo apt-get install gdebi-core

$ sudo wget

$ sudo gdebi shiny-server-

$ sudo start shiny-server

$ sudo status shiny-server


Tuesday, April 26, 2016

Generate word cloud in R

TEXT (.txt)

This working script was tested on R 3.2.5. Code is adapted from It reads a text file, processes it to remove unnecessary words and plots it.



filetxt <- "C:\\Users\\310211146\\Documents\\Other\\May_Report.txt"

txt <- readLines(filetxt)
txt <- tolower(txt)
txt <- removeWords(txt, c("\\f", stopwords()))

corpus <- Corpus(VectorSource(txt))
corpus <- tm_map(corpus, removePunctuation)

tdm <- TermDocumentMatrix(corpus)

m <- as.matrix(tdm)

d <- data.frame(freq = sort(rowSums(m), decreasing = TRUE))
d$stem <- wordStem(row.names(d), language = "english")
d$word <- row.names(d)
d <- d[nchar(row.names(d)) < 20,]

agg_freq <- aggregate(freq ~ stem, data = d, sum)
agg_word <- aggregate(word ~ stem, data = d, function(x)

d <- cbind(freq = agg_freq[, 2], agg_word)
d <- d[order(d$freq, decreasing = T),]

wordcloud(d$word, d$freq)


Friday, April 22, 2016

Learn R with 5 Video Tutorials

If you are new to R or wants to enhance your basic skills in R and wondering where to get started provided there is lots of free material available online then you have come to right place. I have presented 5 video tutorials that is more than enough to get you started with R and will push you in direction of excelling R.

1. What is R and How to install it?

2. What are data types and variables in R?

Friday, February 12, 2016

R: Count occurrences in vector

If you are having a vector and you want to count frequency of individual member (in other words, occurrences of each element) then you can do as following:

use hist() function:

x <- c(1,3,2,5,4,4,2,2,4,2,4,2,4,6,4,5,2)
 [1] 1 3 2 5 4 4 2 2 4 2 4 2 4 6 4 5 2
or use summary() function:

x <- factor(x)
 [1] 1 3 2 5 4 4 2 2 4 2 4 2 4 6 4 5 2
Levels: 1 2 3 4 5 6
1 2 3 4 5 6 
1 6 1 6 2 1
convert vector into factor and then call summary function.

if you are working with characters then hist() won't be useful as it expects numeric elements so use summary() as shown below:
x <- c("devharsh","trivedi","R","C#","ASP.NET","R","C#","devharsh","R")
x <- factor(x)
 ASP.NET       C# devharsh        R  trivedi 
       1        2        2        3        1 

5.5.1 Authentication Error in Gmail

Code for sending email using gmail in C# :

using System.Net.Mail;
using System.Net;

MailMessage mail = new MailMessage();
mail.From = new MailAddress("");
mail.Subject = "subject";
mail.SubjectEncoding = System.Text.Encoding.UTF8;
mail.Body = "foo foo bar";
mail.Priority = MailPriority.High;

SmtpClient client = new SmtpClient();
client.UseDefaultCredentials = false;
client.Credentials = new NetworkCredential("", "gmail-password");
client.Port = 587;
client.Host = "";
client.EnableSsl = true;


"The SMTP server requires a secure connection or the client was not authenticated. The server response was: 5.5.1 Authentication Required"


Login to your gmail account from which you are trying to send email and then go to

Turn on access for less secure apps and error would be resolved.



To disable deprecation, use _CRT_SECURE_NO_WARNINGS.

Error    1    error C4996: 'sscanf': This function or variable may be unsafe. Consider using sscanf_s instead. To disable deprecation, use _CRT_SECURE_NO_WARNINGS. See online help for details.

Error    1    error C4996: 'fopen': This function or variable may be unsafe. Consider using fopen_s instead. To disable deprecation, use _CRT_SECURE_NO_WARNINGS. See online help for details.


Go to Project -> Properties -> C/C++ -> Preprocessor -> Preprocessor Definitions and add _CRT_SECURE_NO_WARNINGS in the list.

Friday, February 05, 2016

cannot coerce type 'externalptr' to vector of type 'character'


Error in as.vector(x, "character") :
  cannot coerce type 'externalptr' to vector of type 'character'

Problem Code:

con <- dbConnect(RMySQL::MySQL(), dbname = "test")

sql <-
      "INSERT INTO `devharsh` (`un`, `pw`, `bl`) VALUES ('",TextUN,"', '",TextPW,"', '",0,"')"
  dbGetQuery(con, sql)


sql <-
      "INSERT INTO `devharsh` (`un`, `pw`, `bl`) VALUES ('",TextUN$getText(),"', '",TextPW$getText(),"', '",0,"')"


It was giving error because it was not getting text from textbox that we should get by getText() function.

CPU and Memory Usage in C#

using System;
using System.Management;

namespace WMISample
    public class MyWMIQuery
        public static void Main()
                ManagementObjectSearcher searcher =
                    new ManagementObjectSearcher("root\\CIMV2",
                                                 "SELECT * FROM Win32_PerfFormattedData_Counters_ProcessorInformation WHERE NOT Name='_Total'");
                ManagementObjectSearcher searcher2 =
                    new ManagementObjectSearcher("root\\CIMV2",
                                                 "SELECT * FROM Win32_OperatingSystem");
                for(int c=0; c<15; c++)
                    for (var i = searcher.Get().GetEnumerator(); i.MoveNext();)
                        string ts = DateTime.Now.ToString("dd-MM-yyyy HH:mm:ss.fffffff");
                        ManagementObject queryObj = (ManagementObject)i.Current;
                        Console.WriteLine("Core {0} running at {1} % on {2}", queryObj["Name"], queryObj["PercentProcessorTime"], ts);
                    for (var i = searcher2.Get().GetEnumerator(); i.MoveNext();)
                        ManagementObject queryObj = (ManagementObject)i.Current;
                        double free = Double.Parse(queryObj["FreePhysicalMemory"].ToString());
                        double total = Double.Parse(queryObj["TotalVisibleMemorySize"].ToString());
                        Console.WriteLine("Amount used: {0} GB", (total - free)/(1024*1024));
                        Console.WriteLine("Percentage used: {0}%", Math.Round(((total - free) / total * 100), 2));
                Console.WriteLine("Press any key to continue . . . ");
            catch (ManagementException e)
                Console.WriteLine("An error occurred while querying for WMI data: " + e.Message);

Tuesday, January 12, 2016

Compare two files with PSPad

You can go to and download PSPad text editor.

To get the difference between two text files open PSPad editor. Then go to Tools menu and select Text Differences then select first option that is Text Diff with This File...

Input string was not in a correct format in ASP.NET/C#

Server Error in '/' Application.
Input string was not in a correct format.
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

Exception Details: System.FormatException: Input string was not in a correct format.

Source Error:

Line 31:         int ppi = Convert.ToInt32(Text1.Text);


I tried to change int ppi = Convert.ToInt32(Text1.Text); statement to int ppi = int.Parse(Text1.Text); and int ppi = Int32.Parse(Text1.Text); and int ppi = Int16.Parse(Text1.Text); but it didn't work.

The reason was I was using a float value and parsing into an integer one, so I change the erroneous statement to the following, and it worked:

float ppi =  float.Parse(Text1.Text);

Friday, January 08, 2016

Data analysis not available in Excel

If you are using MS Excel version older than 2016 and if Data analysis option is not available in Analysis section under Data tab then follow the instructions below.

Data analysis option is not available.

Go to File menu > Options > Add-Ins > Select "Excel Add-ins" in Manage drop-down list that is located at the end of the window and hit go.

Then select Analysis ToolPak and click OK it will add Data Analysis option under Analysis group in Data tab.

This is how it looks like after enabling the add-in.

Wednesday, January 06, 2016

R: 'file' must be a character string or connection

I was using the Shiny app in R, and I was trying to read a file using fileInput control, but I faced the following error:
Listening on
Error in read.table(file = file, header = header, sep = sep, quote = quote,  :
  'file' must be a character string or connection

Problem code:

file <- read.csv(file=input$file1,head=TRUE,sep=",", stringsAsFactors=FALSE)
csv <- str(file)


inFile <- input$file1
if (is.null(inFile))

csv <- read.csv(inFile$datapath, header=TRUE, sep=",")

The resolution was to use the datapath property of the input file and not the file itself.