Steps to Generate Dynamic Sitemap
So far we have discussed creating a sitemap with static URLs. But what if the number of URLs and the content of existing URLs change from time to time. Suppose we are creating a GFG clone. So, our sitemap should contain URLs of all the articles and the important pages.
For this, we will send a sitemap file from our backend by first looping through all the required records from our database. After this, we will manually add the URLs of other important pages like about, contact, etc.
Step 1: For demonstration purposes, We are considering a basic project with three files.
Javascript
// Filename - App.js const express = require( "express" ), mongoose = require( "mongoose" ), todoRouter = require( "./routers/todoRouter" ), app = express(); app.use( "/" , todoRouter); const port = 3000, db = 'mongodb://localhost/todo' ; mongoose .connect(db) .then(conn => { console.log(`${db} connected`); }); app.listen(port, () => console.log( `Server listening on ${port}`)); |
Javascript
// Filename - model.js const mongoose = require( "mongoose" ), todoSchema = new mongoose.Schema( { title: { type: String, unique: true }, }, { timestamps: true } ); module.exports = mongoose.model( "Todo" , todoSchema); |
Javascript
// Filename - todoRouter.js const express = require( "express" ), router = express.Router(); /* Todo Controller functions */ module.exports = router; |
Step 2: Install the’ sitemap’ package to stream the sitemap buffer and write its data.
npm i sitemap
Step 3: Create the sitemapRouter.js file in your router’s directory. At this point, your folder structure will look like this.
The updated dependencies in package.json file will look like.
"dependencies": {
"express": "^4.18.2",
"mongoose": "^8.0.0",
"sitemap": "^7.1.1"
}
Example: This example implements sitemap with rewuired haders and including required URLs.
Javascript
// Filename - sitemapRouter.js const express = require( "express" ), { SitemapStream, streamToPromise } = require( 'sitemap' ), Todo = require( "../models/todoModel" ), date = new Date().toISOString(), zlib = require( "zlib" ), router = express.Router(); let sitemap; router.get( '/' , async function (req, res) { res.header( 'Content-Type' , 'application/xml' ); res.header( 'Content-Encoding' , 'gzip' ); // If we have a cached entry send it if (sitemap) return res.send(sitemap) try { // Fetching todo records and mapping // it the desired URL pattern const data = await Todo.find(), todos = data.map(({ title }) => `/todo/${title}`), // Base url of our site smStream = new SitemapStream({ hostname: 'https://demosite.com/' }), pipeline = smStream.pipe(zlib.createGzip()); // Write todo URL to the stream todos.forEach( item => smStream.write({ url: item, lastmod: date, changefreq: 'daily' , priority: 0.7 })); // Manually add all the other important URLs smStream.write({ url: '/about' , lastmod: date, changefreq: 'monthly' , priority: 0.9 }) smStream.write({ url: '/contact' , lastmod: date, changefreq: 'monthly' , priority: 0.9 }) // Cache the response streamToPromise(pipeline).then(sm => sitemap = sm); smStream.end() // Stream write the response pipeline.pipe(res).on( 'error' , e => { throw e }); } catch (err) { console.error(err) res.status(500).end() } }); module.exports = router; |
Output: Verify the changes in output by visiting {basename}/sitemap.xml.
Add the Sitemap to your robots.txt file.
User-agent: *
Allow: /
Sitemap: https://baseurl/sitemap.xml
How to Generate Sitemap in a MERN Application ?
Generate Sitemap in a MERN Application refers to creating a file that lists the pages, videos, and other files on your website, as well as their relationships. Search engines (such as Google, Duckduckgo, Bing, and others) use this file to help them crawl your site more efficiently.
Contact Us