Node.js APIs are inconvenient to use with URL strings

[GeoffreyBooth]

Problem

Spinning off from #48740 (comment), many of our APIs such as fs.readFile accept URL instances (what you get from new URL) but not URL strings (like import.meta.url, or the return value of the soon-to-be-unflagged import.meta.resolve). In the case of many (all?) of these, all strings are interpreted as paths. This is frustrating since in ESM, we have easy access to URL strings such as import.meta.url but path strings require using helpers such as fileURLToPath.

Original Idea

Please see discussion in the full thread below; while this first idea kicked us off, we’re still brainstorming other solutions. If/when we reach a consensus on ideas that seem worth implementing, I’ll update this top post with a summary.

Wherever feasible, all Node.js APIs that can accept URL strings should do so. In particular this is most relevant to the fs APIs, especially the ones that already accept URL instances. To avoid ambiguity with path strings, such APIs should only interpret URL strings that begin with file:.

I presume that this would be a semver-major change, to avoid needing to first check for the existence of a file or folder named file: in the local path; or perhaps we could add such a check now in order to land this and backport it, and remove such a check in a semver major.

We would also need to consider the security implications. Per @aduh95:

the fact is that readFile('file:///etc/passwd') currently throws (unless there is a local folder named file:, but that’s unlikely), it would concern me if we were to release a new version of node where it no longer throws.

I don’t really see how this is a security concern, but I concede that there might be issues to consider. Perhaps some can be addressed via permissions or policies. I do feel however that since URL strings are so prevalent in ESM, we should require a high bar for security concerns to outweigh usability for this feature.

cc @nodejs/loaders @nodejs/modules @nodejs/security

[Qard]

The only part of ESM that consistently has a URL that I'm aware of is import.meta.url. Specifiers can accept URLs, but the vast majority of uses are path strings. I don't see how changing everything, and in a knowingly backwards-incompatible way, would be preferable to just adding a path string to import.meta alongside the URL that's already there. 🤷🏻

File URLs are not paths and I don't think they ever should have been treated as such. The absolute nature is verbose, which has poor usability, and putting them in the URL class conflates structural validation of the URL format with correctness of the path string which is inaccurate. There are valid paths on some systems which are not valid URLs.

I'm 👎🏻 on this. I feel even supporting URL instances in the fs API was a mistake. If we wanted a typed representation we should have proposed a Path type as an equivalent to URL specifically for file paths.

[tniessen]

@GeoffreyBooth In the very example you gave, @aduh95 explained that file: may refer to a directory within the process's current working directory. Are you suggesting an ambiguous interpretation of such paths? If so, that seems like something we absolutely should not do.

[GeoffreyBooth]

I don’t see how changing everything, and in a knowingly backwards-incompatible way, would be preferable to just adding a path string to import.meta alongside the URL that’s already there. 🤷🏻

With regard to the second point, I would be happy to add a path string to import.meta if we can get a WinterCG-blessed API to do so. Whether or not we add that doesn’t affect whether or not we make the change I propose here; we can do both.

As for “changing everything,” currently readFile‘s first parameter accepts all the following forms of input: string, Buffer, URL, or FileHandle. It would still allow all those forms of input. Currently, string input throws if it’s a file URL (unless the user happens to have a folder or file literally named file:, with a colon in the name). What I’m proposing here is that it no longer throw, and use the presence of file: as the first five characters of the string as the method to disambiguate paths from URL strings.

In the very example you gave, @aduh95 explained that file: may refer to a directory within the process’s current working directory. Are you suggesting an ambiguous interpretation of such paths? If so, that seems like something we absolutely should not do.

In Node today it might refer literally to a folder named file:, with a colon, yes. In my proposal, if a user actually has such a folder, they can continue to reference it as a path by prepending ./ to it or making it an absolute path.

I’m not proposing making the string parsing ambiguous. We have two options for how to handle disambiguation, and we would document what method we’re choosing:

1. Strings beginning with file: are always interpreted as URL strings and that’s it. Everything else is a path, like today.

2. When parsing a string beginning with file:, Node would first check to see if a file or folder named file: existed on disk. If it does, treat the string as a path; otherwise, treat the string as an URL string. Strings not beginning with file: are paths, like today.

The first option would be a semver-major change, because of the extreme edge case of a file or folder actually named file:. The second option could land today and get backported, unless people think that changing the current “throw on URL string” behavior would be breaking to change. I would suggest that even if we land and backport the second option, we switch to the first option in the next major; or we could just go straight to option 1 as a semver major.

[targos]

I'm not convinced this option would help. People don't want (or very rarely) to readFile(import.meta.url).
They want to manipulate paths with the path module first (for example to read a data file relatively to the module's path).
For that, they have to convert the file URL string to a path string

[GeoffreyBooth]

People don’t want (or very rarely) to readFile(import.meta.url).

The kind of use case I had in mind was something like readFile(new URL('.', import.meta.url) + 'config/app.json'), essentially the same as readFile(join(__dirname, 'config/app.json')). You wouldn’t need join because URLs are always forward slashes regardless of operating system. It adds a bit more predictability to working with references to files, and it lets us avoid needing any helpers at all, whether join or fileURLToPath.

[LiviaMedeiros]

A directory name equal or starting with file: is something that easily occurs in a situation where directory structure reflects remote paths, hence things like fs.mkdir('file:///etc/npm', { recursive: true }) or fs.rm('file:///etc', { recursive: true }) switching from ./file:/ base to / might be too much of a breaking change.
We also can't have meaningful check if file with such name exists for methods that work with paths that might not exist yet (e.g. writeFile).

This is frustrating since in ESM, we have easy access to URL strings such as import.meta.url but path strings require using helpers such as fileURLToPath.

For APIs that accept URL instances, we can use new URL(import.meta.url) instead of path strings.

readFile(new URL('.', import.meta.url) + 'config/app.json')

The correct way here is readFile(new URL('config/app.json', import.meta.url))

Overall, I think the directions of improvement might be:

• making node:path APIs accept with URL instances, converting them to absolute paths
• Path class that would guarantee its content to be a valid path (e.g. existing methods can construct a path with NUL character or path longer than PATH_MAX without any warning) and have methods like Path.prototype.toURL() and Path.fromURL()
• helper methods somewhere that would make path-specific parts (extension, basename, dirname, relative path, etc.) work with URL
• if the namespace like import.meta.node gets standardized, maybe import.meta.node.URL with frozen URL instance rather than string
• fetch('file:///path/to/local/file') that would make reading operations protocol-agnostic

[aduh95]

• fetch('file:///path/to/local/file') that would make reading operations protocol-agnostic

+1 to that, related: wintercg/fetch#5

[ShogunPanda]

2. When parsing a string beginning with file:, Node would first check to see if a file or folder named file: existed on disk. If it does, treat the string as a path; otherwise, treat the string as an URL string. Strings not beginning with file: are paths, like today.

+1 to that.

I also think adding the string version of import.meta.url (for instance named import.meta.path) would also help a very lot.