Getting meaningful content by using web embedded content offerings

  • These are some domains/platforms which are easily used...

                                                                                                                                   
    All trademarks, logos and brand names are the property of their respective owners. All company, product and service names used in this website are for identification purposes only. Use of these names,trademarks and brands does not imply endorsement.
  • Typically these return a string which contains iframe or blockquote HTML code, which often has hard-coded properties for such things as height and/or width. We need to change those values if we want to use these embedded offerings. That calls for 'virtual DOM' (see below)
  • It is important to note that you can only change HTML properties with the intent of using this content 'as is' -- without any content changes/deletions/redirects of any kind. For instance, it would be breaking the terms to change the value of the "title" attribute, or any value within the 'blockquote' innerHTML.


Assume you have a string of iframe or blockquote HTML code from one or more of the above platforms...
<iframe src="https://www.npr.org/player/embed/1086537324/1086556909" width="100%" 
height="200" title="NPR embedded audio player"></iframe>
<blockquote class="twitter-tweet"><p lang="en" dir="ltr">
Where in Maine did you explore this weekend? 🌊 <a href="https://twitter.com/hashtag/MaineThing?src=hash&ref_src=twsrc%5Etfw">
#MaineThing</a> <a href="https://twitter.com/hashtag/optoutside?src=hash&ref_src=twsrc%5Etfw">#optoutside</a> 
<br><br>📸 via Instagram: wicked_awesome_maine <br>📍: Roque Bluffs <a href="https://t.co/F018CoNifA">pic.twitter.com/F018CoNifA</a></p>— 
Maine Tourism Office (@visitmaine) 
<a href="https://twitter.com/visitmaine/status/1503399622152175620?ref_src=twsrc%5Etfw">March 14, 2022</a>
</blockquote><script async src="https://platform.twitter.com/widgets.js" charset="utf-8"></script>
                //
                // virtual DOM
                //
                // Note: there are perhaps more direct and easier ways to do this, but this code is 
                // provided to show how any html element(s) can be manipulated and re-used in many ways
                // without screwing the whole thing up
                //
                var iframeString = [string from platform]
                var tempframe = document.createElement("iframe")
                tempframe.id = "t1";
                document.body.appendChild(tempframe);
                var y = tempframe.contentWindow || tempframe.contentDocument;
                if (y.document) y = y.document;
                y.write(iframeString)
                y.location.reload(); // now it is ready to use
                // now we can do whatever we want to the new DOM
                var elements = y.getElementsByTagName("iframe");  // this could also be 'blockquote'
                for (var i = 0; i < elements.length; i++) {
                    elements[i].style.marginTop = "2%";
                    elements[i].style.display = "block"
                    elements[i].style.width = "900px"
                    elements[i].style.height = "auto"
                }
                y.location.reload();
                var completedHTml = y.body.innerHTML; 
                var elem = document.getElementById("t1")  // remove temp iframe
                elem.parentNode.removeChild(elem);
                

Getting meaningful semantic information by parsing the structures of web pages.

Semantic data is stored within HTML pages using several different methods. Typically more than one method is used within a given page. These methods are:
  • ld+json
  • microdata
  • Open Graph
  • HTML header meta tags
  • HTML5 Semantic tags
Semantic data is defined using a framework stored at Schema.org. Schema.org is an organization which regulates and defines a heirarchy of semantically named items and structures.

All browsers that have broadly-based usage support semantic data.

Why use semantic data?
It is in the interest of web sites to include this semantic data if they wish to be found and used by their customers using search engines such as Google, Bing. It is in the interest of search engines to have this data so they can produce meaningful results for the user.

In addition, there are other companies who have services (i.e., not search engines) that use this data. Some examples of those where they directly use the semantic structure at schema.org:
  • an audio internet appliance that speaks recipe ingredients and instructions to you, and steps you through the process. Click here to see this semantic definition.
  • tourism and hospitality providers that return restaurant opening hours, a menu, location, access to reservations, customer ratings, price range, etc, click here
  • retail middle-men who have customers that are buying a product and want to compare similar products and prices, to see user ratings, user reviews, product attributes, photos, etc, click here
  • Bookclub applications click here

Working with semantic coding methods

ld+json

  • "ld" stands for "Linked Data". Within the HTML code of a web page this linked data, using named definitions from that schema, is stored in json format. It can be found within a "script" tag:
    <script type="application/ld+json">
    json data
    </script>
  • json is a protocol used to efficiently store structured data in a textual format.
  • a developer just has to identify the named entity @type and then instantiate and populate the object class which maps to the proper schema entity -- viola, real semantics! Here is an example -- a 'Recipe'' type from this web site:
    Interesting tidbit: this has 'how-to' nodes -- these nodes are used by Amazon Alexa and other platforms to provide step-by-step spoken instructions in your kitchen

  • JSON data:
    {
        "@context": "http:\/\/schema.org",
        "@type": "Recipe",
        "name": "Pesto Mushroom Bolognese Lasagna",
        "author": {
        "@type": "Person",
        "name": "mandyolive.com"
        },
        "datePublished": "2020-05-05",
        "recipeYield": 6,
        "description": "A twist on this slow-cooked Italian-style favorite. This mushroom bolognese style sauce is laced between sheets of lasagna, creamy pesto ricotta, and topped with gooey cheese. \ufeff",
        "image": [
        "https:\/\/www.mandyolive.com\/wp-content\/uploads\/2021\/10\/pesto-mushroom-bolognese-lasagna-feature-photo.png"
        ],
        "prepTime": "PT30M",
        "cookTime": "PT1H15M",
        "performTime": "PT1H15M",
        "totalTime": "PT1H45M",
        "recipeIngredient": [
        "1 pound fresh mushrooms, chopped",
        "1 fennel bulb (stalks + fronds if available), diced.\u00a0",
        "1 large carrot, peeled + grated (about 1\/3 c.)",
        "4 cloves of garlic, grated",
        "1 cup dry white wine",
        "1 (28 ounce) can diced or crushed tomatoes",
        "1\/4 cup tomato paste",
        "2 teaspoons dried herbs",
        "Salt + Pepper",
        "2 cups whole milk ricotta cheese",
        "1 cup whole milk\u00a0",
        "1 box lasagna noodles",
        "2 cups Fontina or Provolone cheese",
        "Fresh mozzarella, torn",
        "Freshly grated Parmesan or Asiago cheese",
        "Flat leaf parsley, finely chopped\u00a0",
        "2 cups mixture of power greens salad blend, (baby kale, arugula, spinach, swiss chard), flat-leaf parsley, and fennel fronds.\u00a0",
        "2 tablespoons nuts or seeds, I used almonds",
        "1 clove garlic",
        "1\/2 cup Parmesan",
        "1\/2 cup good quality olive oil"
        ],
        "recipeInstructions": [
        {
            "@type": "HowToStep",
            "text": "Add all ingredients, except for the olive oil, into a food processor. Pulse the mixture while slowly adding olive oil (you may not use all of it). The greens should be finely chopped, and the consistency of the pesto should be thick but not paste-like.",
            "position": 1,
            "name": "Add all ingredients, except for the olive oil,...",
            "url": "https:\/\/www.mandyolive.com\/pesto-mushroom-bolognese-lasagna\/#mv_create_64_1"
        },
        {
            "@type": "HowToStep",
            "text": "Once the pesto is finished, mix it with the ricotta cheese and set it aside.",
            "position": 2,
            "name": "Once the pesto is finished, mix it with...",
            "url": "https:\/\/www.mandyolive.com\/pesto-mushroom-bolognese-lasagna\/#mv_create_64_2"
        },
        {
            "@type": "HowToStep",
            "text": "Place the lasagna noodles in a large 9 x 13-inch baking dish, pour enough hot water over to cover. Set aside for 30 minutes or until soft and pliable. Drain the noodles, separate them, and set aside.",
            "position": 3,
            "name": "Place the lasagna noodles in a large 9...",
            "url": "https:\/\/www.mandyolive.com\/pesto-mushroom-bolognese-lasagna\/#mv_create_64_3"
        },
        {
            "@type": "HowToStep",
            "text": "Clean the mushrooms with a wet paper towel to remove any residual dirt. Dry thoroughly with a clean kitchen towel or additional paper towels, then quarter, or thick-slice, each mushroom. You can chop them a bit smaller if they are on the larger side.",
            "position": 4,
            "name": "Clean the mushrooms with a wet paper towel...",
            "url": "https:\/\/www.mandyolive.com\/pesto-mushroom-bolognese-lasagna\/#mv_create_64_4"
        },
        {
            "@type": "HowToStep",
            "text": "Heat a 4.5 qt. dutch oven over medium-high heat and oil with 2 tablespoons of olive oil. Sear the first side of the mushroom for about two minutes. Try not to move them around. Just leave them to brown. Once the initial side is golden brown and some of the liquid has released, toss and sear the other sides--about 1 - 2 more minutes more.",
            "position": 5,
            "name": "Heat a 4.5 qt. dutch oven over medium-high...",
            "url": "https:\/\/www.mandyolive.com\/pesto-mushroom-bolognese-lasagna\/#mv_create_64_5"
        },
        {
            "@type": "HowToStep",
            "text": "Next, add an additional tablespoon of oil if needed. Lower the heat to medium-low, add the grated carrots, diced fennel bulb, and stalks (if using). Season with dried herbs, and a large pinch of salt and pepper. Saute until translucent and tender-- approx. 10-15 mins. Then add garlic. Cook the vegetables tossing frequently for another 1 - 2 minutes.",
            "position": 6,
            "name": "Next, add an additional tablespoon of oil if...",
            "url": "https:\/\/www.mandyolive.com\/pesto-mushroom-bolognese-lasagna\/#mv_create_64_6"
        },
        {
            "@type": "HowToStep",
            "text": "Once the mushrooms have finished browning and have reduced in size by half, the vegetables have softened, and the garlic is fragrant, deglaze the pan with white wine. Scraping the bottom.",
            "position": 7,
            "name": "Once the mushrooms have finished browning and have...",
            "url": "https:\/\/www.mandyolive.com\/pesto-mushroom-bolognese-lasagna\/#mv_create_64_7"
        },
        {
            "@type": "HowToStep",
            "text": "Next, add the tomatoes (with juice), tomato paste, and milk. Stirring to combine. Simmer the sauce over medium-low heat for at least 30 minutes. The sauce should just barely bubble. Stirring occasionally while scraping the bottom of the pot. The consistency of Bolognese is on the thicker side, so when the sauce is done, a little more than half of the liquid should be evaporated.",
            "position": 8,
            "name": "Next, add the tomatoes (with juice), tomato paste,...",
            "url": "https:\/\/www.mandyolive.com\/pesto-mushroom-bolognese-lasagna\/#mv_create_64_8"
        },
        {
            "@type": "HowToStep",
            "text": "Preheat the oven to 375 degrees F, Grease a 9x13 inch baking dish with butter or olive oil, then spread 1\/4 of the sauce onto the bottom of the pan. Alternate layers of lasagna sheets, pesto ricotta cheese, bolognese sauce, shredded cheeses, and any leftover salad greens. After, top the finished lasagna with remaining bolognese sauce and remaining shredded cheese. Top the pesto mushroom bolognese lasagna with torn pieces of mozzarella and grated parmesan or asiago.",
            "position": 9,
            "name": "Preheat the oven to 375 degrees F, Grease...",
            "url": "https:\/\/www.mandyolive.com\/pesto-mushroom-bolognese-lasagna\/#mv_create_64_9"
        },
        {
            "@type": "HowToStep",
            "text": "Bake loosely covered with aluminum foil for 30 minutes or until the lasagna has cooked through and the top is bubbly. Remove the foil and cook for an additional 15 minutes until the lasagna is golden brown. Additionally, you can hit it with the broiler to help brown the top.",
            "position": 10,
            "name": "Bake loosely covered with aluminum foil for 30...",
            "url": "https:\/\/www.mandyolive.com\/pesto-mushroom-bolognese-lasagna\/#mv_create_64_10"
        },
        {
            "@type": "HowToStep",
            "text": "Garnish with fresh, flat-leaf parsley and fennel fronds. Let stand for 10 minutes before serving.",
            "position": 11,
            "name": "Garnish with fresh, flat-leaf parsley and fennel fronds....",
            "url": "https:\/\/www.mandyolive.com\/pesto-mushroom-bolognese-lasagna\/#mv_create_64_11"
        }
        ],
        "keywords": "pesto bolognese lasagna, mushroom bolognese, meatless lasagna recipes, spinach lasagna recipes, vegetable lasagna recipes, fennel recipes, what to do with fennel bulb, pesto mushroom bolognese lasagna, easy lasagna recipe, easy pesto recipes, how to make pesto, what to do with pesto, mushroom lasagna recipes, vegetarian lasagna recipes, how to make lasagna, the best lasagna recipes, no-boil pasta recipes, pesto pasta recipes, recipes for a crowd, pasta recipes, dinner recipes, easy pasta sauce recipes,  homemade lasagna recipes, how to make lasagna, ",
        "nutrition": {
        "@type": "NutritionInformation",
        "calories": "542 calories",
        "carbohydrateContent": "21 grams carbohydrates",
        "cholesterolContent": "82 milligrams cholesterol",
        "fatContent": "39 grams fat",
        "fiberContent": "5 grams fiber",
        "proteinContent": "25 grams protein",
        "saturatedFatContent": "16 grams saturated fat",
        "servingSize": "1",
        "sodiumContent": "723 grams sodium",
        "sugarContent": "8 grams sugar",
        "transFatContent": "0 grams trans fat",
        "unsaturatedFatContent": "21 grams unsaturated fat"
        },
        "url": "https:\/\/www.mandyolive.com\/pesto-mushroom-bolognese-lasagna\/"
    }
    

Microdata

Microdata is a bit older than ld+json, it also is defined using a framework found at Schema.org. Microdata items are stored within HTML elements (i.e., <div>, <span>, <meta> etc.) inserted as an attribute named "itemprop". That attribute:
  • stores the name of the schema item, and
  • is accompanied by a "content" attribute which stores the value of the schema item.

Open Graph

Facebook's Open Graph stores its items within meta tags. When someone posts a web link to Facebook, Facebook will get at a minimum three of those meta tags from that link
When Facebook initially released Open Graph 50,000 web sites had added those meta tags within one week. This has become the de facto standard for most web sites. Almost all web sites now have at least these 3 metatags. Shows the economic and social power of semantics in linking items.

Meta tags

The oldest HTML tags for storing and exposing metadata within a web page are 'meta' tags. These are found in the HTML header. These <meta> HTML elements are typically used to store some information such as the character set used on the page, the keywords of the document, the viewport, the author, and many other provider-specific implementations (such as Facebook Open Graph tags.)

HTML Semantic tags


Other sources:

Some websites also provide 'keyword' tags embedded within their semantic data. These keywords can be single words or phrases of words.


** A note about semantic data for product price and availability:
If you extract the price and availability for a product from a web page (both ld+json and microdata frameworks provide this information in a structured way) and intend to produce the data for your own purposes, please note that many businesses require that you cannot just get it once and let it remain static. You must be able to dynamically refresh that price and availability anytime you 'offer' it. Schema.org even uses the topographic term of "offer" for this semantic entity. Businesses such as Target, Amazon and Bestbuy require this, and they provide a web service to get that updated information. An example is Target's "Redsky" service.



Important:
If an image, text or link on a web page is retrieved via the above frameworks or via RSS you can assume the owner of that webpage has decided to allow those assets to be freely distributed. A typical reason for this would be a posting of that item on Facebook, Instagram, etc. But otherwise you must assume all other assets are copyrighted by the owner of that webpage, and their rights should be respected.



Contact me