How to get data from NYT developer network using python

Ok so here are the steps:

Step1: Go to NYT developer network. Request an API key. There are many kinds of API’s available. I’ll go with Article Search API for now.

Step2: Go to the Article Search API page, and click on to Read Me on the top right. Scroll down to “Requests” section and copy the url with q=new+york+times.

Step3: Replace the ending ### with the key you got in the mail and open the link. You’ll see something like this:

Screen Shot 2016-10-08 at 11.57.54 PM.png

Step4: You will see all the data that was exposed by the API. Download JSON Formatter extension of chrome to see a readable version of the API data. Here’s how it should look:

Screen Shot 2016-10-08 at 11.58.37 PM.png

Step5: Look at the formatted data for a second. If you collapse all you see that first of all, this is an object that starts with { and ends with }. Secondly, you’ll see that there are three main elements in this object; response, status and copyright. We’ll expand each of these one by one to understand what lies inside and where the actual data about articles is. Later on we’ll do this using python.

Step6: So inside there are two objects; meta and docs. Inside the docs object, there are 10 objects. Each object is an article. Each article has several properties like web_url, snippets, lead_paragraph, headline etc. Some of these elements are objects themselves, some are arrays, some strings, some numbers. So that’s it, we have seen and understood the data we got from NYT. It’s structured this way because we can use code to extract data from it easily and use it for our purposes. So lets see how can we do that.

Step7: Open terminal and install pip. Use the command sudo easy_install pip. You can be in any directory while you do this. We need pip because we will have to install several libraries in order to extract data from the JSON using Python.

Step8: Google search for how to extract data from JSON using python. You’ll find this stackoverflow example. We will use the code given here to extract data from our JSON using python.

Step9: Open Sublime-text and save the file as nyt.py. DOTpy is the extension for python. In the terminal, change your directory to the folder where you saved the file.

Step10: Now paste the code we found in stackoverflow. We will start by just using this:

import json, requests

url = ‘http://api.nytimes.com/svc/search/v2/articlesearch.json?q=new+york+times&page=2&sort=oldest&api-key=2310ba05bce344d98f720ae433ff8e5b’

resp = requests.get(url)
data = json.loads(resp.text)

What this does is that it imports the json and requests library. In url, we need to put the url we used to access our NYT data earlier in Step4. Next we use the get function in requests library to store data in a local variable called resp. This gives us everything in the form of a continuous string. We need to change this string into json format. So we use the loads function of json library to do that and store the json data into a variable called data.

Step11: We cant import the json and requests library unless we install them. So we’ll install these libraries using pip. In terminal, write “sudo pip install requests” and “sudo pip install json”.

Step 12: Now we are ready to parse the data using python and use the terminal to see the output of our requests. To run a simple command, we can say:

print data

in our python file. Then we can go to the terminal and say:

python nyt.py

Make sure you are in the same folder in terminal where the python file is. What the above command should do is produce the entire data we got from the api, just like we saw in the browser window without the JSON formatter. Here’s how it would look:

Screen Shot 2016-10-08 at 11.56.15 PM.png

Again a single string of data. You can clear that using cmd+K.

Step13: We can do something about the appearance of this data. We can use the pprint (called pretty print) command from the pprint library. To do that write this with the import commands on top:

from pprint import pprint

now we can use pprint(data) instead of print data. When we do python nyt.py in the terminal now we get a better formatted form of data like this:

Screen Shot 2016-10-09 at 12.05.42 AM.png

Step14: Now we want to start extracting data from it one by one, just like using the expand collapse in the browser. So for that we use the for loop in python, like this:

for key in data:

print key

What this does is prints all the keys it found in the object data. As we saw in the browser, there are three keys; status, response and copyright. When we do python nyt.py in our terminal, we should get the same thing.

Screen Shot 2016-10-09 at 12.11.09 AM.png

Step15: To get the data from each of these three keys we use this:

print data[“response”]

or

pprint(data[“response”])

This should give the data inside response object, which is inside data object. Like this:

Screen Shot 2016-10-09 at 12.17.56 AM.png

Step16: We can go like this again and again. So we can see all the keys inside response by:

for key in data[“response”]:

print key

and then

print data[“response”][“docs”]

and so on.

Step17: If we see docs, it is not an object, it is an array. We know this because it starts and ends with [] rather than {}. If we print the first element of this array by

print data[“response”][“docs”][0]

or

pprint(data[“response”][“docs”][0])

we see that it is data on an article.

Step18: Now that we have come to the list of articles and we see that there is an object called headline that has a key called main which has its value as the actual sentence of headline of the article, how do we print all the headlines of all the articles that we have? here’s how:

for key in data[“response”][“docs”]:

print key[“headline”][“main”]

This will print all the headlines when we do python nyt.py in our terminal.

Step19: How to query this database now? Remember the URL? Here how it looks:

http://api.nytimes.com/svc/search/v2/articlesearch.json?q=new+york+times&page=2&sort=oldest&api-key=###

the q=new+york+times is the query part.

so what we can do is this:

query = “Presidential”

url = “http://api.nytimes.com/svc/search/v2/articlesearch.json?q=” + query + “&page=2&sort=oldest&api-key=###”

This will search the database for query. Suppose we want all the headings with that query in them we just do what we did before:

for key in data[“response”][“docs”]:

print key[“headline”][“main”]

Step20: Lastly, what of we want to query using terminal and not change the code all the time. For example what if we want to do something like this in terminal:

python nyt.py “query”

To do this we include a library called sys by

import sys

Now if we write print sys.argv in our python code and do python nyt.py in terminal, we get nyt.py in return. What this means is that sys.argv (argv= argument value) is an array that has different words in the command given in the terminal as its elements. So if we want to do a query by doing python nyt.py “query” we need to pass sys.argv[1] in the url in our python code.

So we do this:

import requests
import json
from pprint import pprint
import sys

query= sys.argv[1]

url = “http://api.nytimes.com/svc/search/v2/articlesearch.json?q=” + query + “&page=2&sort=oldest&api-key=2310ba05bce344d98f720ae433ff8e5b”
resp = requests.get(url)
data = json.loads(resp.text)

for key in data[“response”][“docs”]:
print key[“headline”][“main”]

That’s it for now.

Advertisements
How to get data from NYT developer network using python

How to handle images in RWD

Relative Image Size

img{

width: 100%;

}

Image occupies 100% width of the browser, no matter its natural width. This may lead to pixelation.

img{

max-width: 100%;

}

Images scale to occupy full width of the browser only till their maximum natural widths.

Similarly

img{

max-width: 50%;

}

Image will scale to occupy 50% of the viewport size until its max natural width is achieved.

Now, there is also a neat way to give relative widths to images put side by side while giving some padding in between them by “calc” function:

body{

font-size: 0;

}

img {

width: calc((100% – 20px)/3);

margin-right: 10px;

}

img:last-of-type {

margin-right: 0;

}

This code will put three images side by side while leaving a 10px margin between them. it is important to cancel out the inherent space between inline-block elements by putting body font-size:0. Also it is important to properly follow the syntax on calc i.e. leaving space around that minus sign.

Two other relative units are vmin and vmax.

img{

width: 100vmin;

}

The width of image will be 100% of whatever is less among width or height of the viewport. The height of image will be decided according to the aspect ratio.

similarly

img{

width: 100vmax;

}

The width of image will be 100% of whatever is more among width or height of the viewport.


Image Formats

  • Use jpg or webP formats for photographic images
  • Use SVG whenever you can for logos and such. If not, use PNG which is better than GIF in terms of colours, compression and no licensing issues

Size and compression

First rule to show images on a range of devices is to keep the “Total bits” minimum for that device.

Total Bits = Number of Pixels (i.e. size of the image) x bits per pixel (i.e. compression of the image)

Using grunt:

http://24ways.org/2013/grunt-is-not-weird-and-hard/

In order to check if we are serving the minimum possible size of images to different devices, we can use PageSpeed Insights:

https://developers.google.com/speed/pagespeed/insights/


Loading Responsively

Letting the browser decide which image to load based on the environment in which the images are loading rather than deciding at build time using media queries.

Method 1

Now the browser knows two things:

  1. The DPI of the screen
  2. The screen size

To appropriately choose which image to load it needs to know two more things:

  1. The natural size of the images.
  2. The display size of the images.

To do this we can use the “srcset” and the “size” attribute of img tag. Here is how:

<img

     src=”small.jpg”

     srcset=”small.jpg 500w, medium.jpg 1000w, large.jpg 1500w”

     sizes=”(max-width: 250px) 100vw, 50vw”

     alt=”Wallaby”

>

So whats happening here is that browser now knows what is the natural width of each image:

small.jpg = 500px wide

medium.jpg= 1000px wide

large.jpg= 1500px wide

and what percentage of screen width does the image has to cover:

100% till 250px viewport size and after that 50%

Hence, instead of we deciding which image to load while writing the code, we have given all the necessary information to the browser to let it decide which image is best suited based on environmental conditions such as device-pixel-ratio, screen sizes, bandwidth etc.

We can also use the calc function in our sizes attribute like this:

sizes=”(min-width:500px) calc((100vw – 12em)/3), 100vw”

Note, in case the browser does not understand the srcset attribute, we have given a fallback with good old “src” attribute.

Method 2

There is an easier way if you want to render fixed width images.

<img srcset=”small.jpg 1x, large.jpg 2x”,

src= “small.jpg”

alt=”A rad wolf”

/>

Here since the display size of image is fixed, we can simply tell the browser to load small.jpg on a 1x display and large.jpg on a 2x display instead of saying:

<img srcset=”small.jpg 320px, large.jpg= 640px” sizes=”200px” alt=”A Rad Wolf” />

In this case the browser will have to see that ok the display size is 200px and I am on a 2x display so I need an image at least 400px wide so I will go for large.jpg. Instead of all that we have simply told browser to load the larger image when he has a 2x display.

Method 3

The picture element. With the previous methods we can control the optimal size of the image for a given environment. Now we want to control which image gets rendered before we calculate the optimal size of that image.

The picture element lets us define different pictures all together for different break-points. This is called art direction.

<picture>

<source

media= “(min-width:360px)”

sizes=”33.33w”

srcset= “small.jpg 320w, medium.jpg 640w, large.jpg 1000w”

/>

<sources

srcset=”cropped-large.jpg 2x, cropped-small.jpg 1x”

/>

<img

src=”small.jpg” alt=”A Rad Wolf”

/>

</picture>

Another customisation that we can do with the picture element is with regard to file format:

<picture>

<sources type=”image/svg” src=”logo.svg” />

<sources type=”image/png” src=”logo.png” />

<img src=”logo.gif” alt=”logo” />

</picture>

That’s kind of how it is done.

How to handle images in RWD

How to use custom grids with gutter space to layout content

Like this:

1. First of all, make all the boxes border-box so the padding and everything stays inside the box.

* {
-webkit-box-sizing: border-box;
-moz-box-sizing: border-box;
box-sizing: border-box;
}
*:before,
*:after {
-webkit-box-sizing: border-box;
-moz-box-sizing: border-box;
box-sizing: border-box;
}

2. Make a container to overall centre your content on the page. If you want your content to not touch the edges of the screen then you can also give a padding to this container.

.container{
max-width: 960px;
margin: auto;
}

3. Make a row. All the content-chunks of different sizes will be floating left inside this row. Row will then be clearfixed. Also, row will offset the padding that first and last columns will have, in order to align the content inside the columns with the content which is not.

.row {
margin-left: -15px;
margin-right: -15px;
}

.row:before,
.row:after {
content:” “;
display: table;
}

.row:after {
clear: both;
}

4. Make columns for content inside the rows. Columns decide the width of each component and they float left.

.col-1,
.col-2,
.col-3,
.col-4,
.col-5,
.col-6,
.col-7,
.col-8,
.col-9,
.col-10,
.col-11,
.col-12 {
padding-left: 15px;
padding-right: 15px;
float: left;
}

.col-12 {
width: 100%;
}
.col-11 {
width: 91.66666667%;
}
.col-10 {
width: 83.33333333%;
}
.col-9 {
width: 75%;
}
.col-8 {
width: 66.66666667%;
}
.col-7 {
width: 58.33333333%;
}
.col-6 {
width: 50%;
}
.col-5 {
width: 41.66666667%;
}
.col-4 {
width: 33.33333333%;
}
.col-3 {
width: 25%;
}
.col-2 {
width: 16.66666667%;
}
.col-1 {
width: 8.33333333%;
}

And thats it. Put everything in columns and layout your content the way you want.

Now there are two other ways of laying out the content:

  1. Using display: inline-block for the columns. You put everything you don’t want to put in columns in a container with padding equivalent to the gutter space that you need between columns.  Then put everything you want to put inside columns in a row with padding half of that. The remaining half will be compensated by the padding inside these columns. Remember that inline-block adds a bit of extra room to comment the spaces in html.
  2. Using display: flex; flex-wrap: wrap on the row element. And width: calc((100%-30px)/3); for the width of each component inside the row. But calc has poor compatibility with browsers and this you cant have varying sizes of components. Along with that put margin-right: 15px; for each component and margin-right:0; for .component:last-of-type.

Fin.

How to use custom grids with gutter space to layout content

How varying pixel densities are managed in responsive

By using the concept of DIPs i.e. Device Independent Pixels. A DIP has an actual physical size for example 1 DIP in android is 1/160 of an inch or 1.5875 mm. This is absolute and independent of any device.

So when you give your sizes in html in pixels you are actually giving them in DIPs. While rendering the browser looks at the dip value of each component and the device pixel ratio or the dppx.

Device Pixel Ratio(dppx) = Number of device pixels/ Number of device independent pixels.

That means ddpx is nothing but how many device pixels are there in one logical pixel (in just one dimension). In 4k screens dppx is 4 i.e. they have managed to put 4 pixels inside one logical pixel.

Therefore the browser can calculate the pixel size on the device, based on the dips in the code and dppx of the device.

Bottomline, you define DIPs in the code without worrying about varying pixel densities of screens.

One twist, you actually write CSS pixels when you code and not DIPs. The CSS pixels first get converted to DIPs and then to actual hardware Device Pixels.

Why? To allow zooming. We can instruct the browser that to begin with 1 CSS px = 1 DIP. But when user zooms the number of CSS pixels defined in “width=320px;” remains constant but the ratio of CSS pixels and DIPs change. For example in case of 2X zoom-in, each CSS pixel becomes 2 DIP, hence “width=320px” becomes 640DIPs which is then converted to actual hardware pixels using the dppx ratio.

How? Using the meta tag:

With initial-scale=1 we are saying that 1DIP=1CSS px. Without this definition, the browsers tend automatically adjust the scale value and resize the page.

Apart from that with width=device-width we are telling the browser to match the screen’s width in DIPs. This allows the content to reflow based on the media-queries. We have to declare it because unless we tell the browser that our design is fit to work on a small screen it assumes it wasn’t and renders the screen as if width=980px (DIPS).

How varying pixel densities are managed in responsive

How responsive works on bootstrap

Ok very quickly, (all this for bootstrap3)

There are three things that work together:

  • Container. To set up whole page, margin:auto, padding left and right: 15px fixed. Width of the container depends on the device size, they have added a media query for min-width at 768, 992, 1200px. (container-fluid has no width definition)
  • Row. This is used whenever you want to create columns. Its job is to off-set the padding set by container for the first and the last columns.
  • Col-xs/sm/md/lg-1-12. These have a fixed left and right padding of 15 px, giving a gutter space of 30px between columns. These columns float left. They have provided a separate clearfix class that adds “” with display “table” before and after a bunch of stuff.

Now, how does responsive work.

On every device the number of columns are 12 and the width of each column is also same in percentage. That means whether on phone or a desktop 12 columns mean 100%, 6 means 50% and so on. Also the padding is fixed, 15px on either side of the column.

What you can do to make it responsive is that you can change the number of columns each component will occupy based on the device size and thats when these xs, sm,md and lg come into picture.

The way they are defined is that in css there is a media query on device size. Below 768 it will read xs and nothing else, below 992px it will read sm, below 1200 it will read md, and above that lg. So in your html you can define the div like this:

<div class=”col-xs-12 col-sm-12 col-md-6 col-lg-4″>….</div>

So what will happen is that on extra small and small screens the component will occupy 12 columns which is 100% as the browser wont even read the other classes because of the media query while defining these classes in css. Similarly in medium it will occupy 6 columns and in large only 4 columns. Since the columns float left they will stack one on top of the other when screen sizes change.

So bottom line. How does responsive work in bootstap?

  • The gutter width is fixed at 30px (15+15) between two columns
  • The column size in percent and number of columns is same irrespective of devices.
  • But based on device size you can change the number of columns that each component would occupy. This happens because of separate xs, sm, md and lg definitions based on min-width media query in css.
How responsive works on bootstrap