APIs, models, pythons??

So, I spent most of the last few hours reading about what APIs do and how they can be useful. Now I'm lost. How do I start using them?

I'm using python & django and desire to pull data from MDPI and/or CrossRef.

I guess I have a bunch of questions.

  • do I need matching models on my end?
  • what format am I getting from them?
  • How do I get the info to move into my database?
  • I keep seeing json and xml, but I'm not sure what this has to do with anything.

Any help is super appreciated. I feel stuck because I don't really know what to google.

Ralf Pitter

You are basically hitting a http endpoint and they are sending you data back. As you mentioned, most of the times json or xml. What you do with these data is fully up to you. You can either store them compketely in your db. Then you need corresponding models, yes. Or you just store some of the values somewhere or you just use them to calculate something. You can basically do anything you want with them. Think about them like user generated data swnt by a form or something line that.

0 Likes
Vicki Langer Author

Okay, so I need corresponding models. I know how to do that.

How do I send and receive​ the request though?

And from there, I assume I have to tell python that their file should be put in my db? somehow

0 Likes
Ralf Pitter

It's really just sending an http request to the corresponding endpoint.

0 Likes

Think of grabbing API data as a separate task than doing something with the result on your end. Web APIs respond to HTTP requests (like GETs or POSTs). They will respond with data and then it's up to you to do what you want with it.

It doesn't automatically get deposited into a database or anything, you will have to save it using conventional methods used in Django. I am not a Django user but here's the relevant docs: https://docs.djangoproject.com/en/2.2/topics/db/models/

So you'd basically create a table/model that corresponds to the data you want to save. In their docs they talk about saving Fred Flinstone, you'd do the same but more according to your own data/models.

0 Likes
jase

Hi @vickilanger Firstly, you're not alone, I remember feeling exactly the same when I first had to connect to an API, staring blankly at docs for what must have been hours…and each API is different. Looking at the examples you provided, as you mentioned, it looks like you can either use MDPI directly which will give you XML back or use CrossRef which will return JSON back. Which you use is really a preference thing.

As these look like public APIs, it doesn't look like you need to worry too much about authentication which is nice, that's another issue on its own.

Rather, using the information from the two sites, here is an example of fetching the agency info of MDPI using CrossRef:

https://api.crossref.org/works/10.3390/s130912044/agency

You can just view that in your browser and it will return a JSON string. While I am not really familiar with Django / Python, no matter the language, the next step is using whatever HTTP client the language / framework has to fetch that same URL and well, see what you GET.

From there, look at how Django can convert JSON into an array or object of some kind that you can then work with, I think this could work:

https://www.django-rest-framework.org/api-guide/parsers/

Play around with that and then see if you can write the returned info out to your console and once you get that right, you should be able to store it in your DB etc.

It's interesting and cool for me that your two options are XML and JSON because there are two main types of APIs, SOAP and REST and usually SOAP uses XML and REST, JSON. Two different methods to achieve the same goal.

With regard to models, XML-SOAP sometimes needs you to create a matching model as the actual data structure is sent as part of the API spec, but this is usually handled by frameworks and helpers and so isn't really an issue anymore. REST API responses are normally parsed from JSON into an array or some JSON object that you can just iterate through. So I don't think you need to worry about matching models up it will probably just be something you iterate through, grab the info you want and assign that to your own model fields for DB storage.

CrossRef's API is mostly RESTful which there is a lot of information on: https://restfulapi.net/

Basically though, it uses different URLs and / or query parameters to send and receive information. What makes it really cool though is that it also uses HTTP methods (GET, POST, PUT, DELETE) to determine actions. So you can request the same URL. Eg. https://someapi.test/apples but depending on how you request it, something different will happen. Traditionally, in this example, GET will fetch a list of apples and POST will make a new apple etc.

Hope that helps a bit!

0 Likes
Vicki Langer Author

@Jase, thank you! That was all super super useful!

I realized my biggest problem is that I had no clue how to read or use the output of the API link.

Once I watched a couple of youtube videos, I think I've got the concept.

I successfully have some data to work with. Now I need to learn how to get it into my database and how to get this API to update on a regular schedule.

0 Likes
Vicki Langer Author

Thanks all the help.

This is something I'll have to write about later because I can't be the only one who has ever had these questions. I didn't know what I didn't know, which was most of my problem.

Turns out you must understand JSON (and sometimes XML) before I even begin to think I could work with an API.

Yes, an API is just a request from an endpoint. That's just the beginning though. I still needed to understand what I need to do with it, can do with it, or that I needs to read docs for each individual API.

0 Likes

Please sign in to leave a comment.