Make default ListingFilesCache table scoped #19616

jizezhang · 2026-01-03T00:11:51Z

Which issue does this PR close?

Builds on feat: add list_files_cache table function for datafusion-cli #19388
Closes Confusing behavior now required to to refresh the files of a listing table #19573

Rationale for this change

This PR explores one way to make ListFilesCache table scoped. A session level cache is still used, but the cache key is made a "table-scoped" path, for which a new struct

pub struct TableScopedPath(pub Option<TableReference>, pub Path);

is defined. TableReference comes from CreateExternalTable passed to ListingTableFactory::create.

Additionally, when a table is dropped, all entries related to a table is dropped by modifying SessionContext::find_and_deregister method.

Testing (change on adding list_files_cache() for cli is included for easier testing).

Testing cache reuse on a single table.

> \object_store_profiling summary
ObjectStore Profile mode set to Summary
> create external table test
stored as parquet
location 's3://overturemaps-us-west-2/release/2025-12-17.0/theme=base/';
0 row(s) fetched. 
Elapsed 14.878 seconds.

Object Store Profiling
Instrumented Object Store: instrument_mode: Summary, inner: AmazonS3(overturemaps-us-west-2)
Summaries:
+-----------+----------+-----------+-----------+-------------+-------------+-------+
| Operation | Metric   | min       | max       | avg         | sum         | count |
+-----------+----------+-----------+-----------+-------------+-------------+-------+
| Get       | duration | 0.030597s | 0.209235s | 0.082396s   | 36.254189s  | 440   |
| Get       | size     | 204782 B  | 857230 B  | 497304.88 B | 218814144 B | 440   |
| List      | duration | 0.192037s | 0.192037s | 0.192037s   | 0.192037s   | 1     |
| List      | size     |           |           |             |             | 1     |
+-----------+----------+-----------+-----------+-------------+-------------+-------+
> select table, path, unnest(metadata_list) from list_files_cache() limit 1;
+-------+---------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| table | path                            | UNNEST(list_files_cache().metadata_list)                                                                                                                                                                                                                  |
+-------+---------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| test  | release/2025-12-17.0/theme=base | {file_path: release/2025-12-17.0/theme=base/type=bathymetry/part-00000-dd0f2f50-b436-4710-996f-f1b06181a3a1-c000.zstd.parquet, file_modified: 2025-12-17T22:32:50, file_size_bytes: 40280159, e_tag: "15090401f8f936c3f83bb498cb99a41d-3", version: NULL} |
+-------+---------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row(s) fetched. 
Elapsed 0.058 seconds.

Object Store Profiling
> select count(*) from test where type = 'infrastructure';
+-----------+
| count(*)  |
+-----------+
| 142969564 |
+-----------+
1 row(s) fetched. 
Elapsed 0.028 seconds.

Object Store Profiling

Test separate cache entries are created for two tables with same path

> create external table test2
stored as parquet
location 's3://overturemaps-us-west-2/release/2025-12-17.0/theme=base/';
0 row(s) fetched. 
Elapsed 14.798 seconds.

Object Store Profiling
Instrumented Object Store: instrument_mode: Summary, inner: AmazonS3(overturemaps-us-west-2)
Summaries:
+-----------+----------+-----------+-----------+-------------+-------------+-------+
| Operation | Metric   | min       | max       | avg         | sum         | count |
+-----------+----------+-----------+-----------+-------------+-------------+-------+
| Get       | duration | 0.030238s | 0.350465s | 0.073670s   | 32.414654s  | 440   |
| Get       | size     | 204782 B  | 857230 B  | 497304.88 B | 218814144 B | 440   |
| List      | duration | 0.133334s | 0.133334s | 0.133334s   | 0.133334s   | 1     |
| List      | size     |           |           |             |             | 1     |
+-----------+----------+-----------+-----------+-------------+-------------+-------+
> select count(*) from test2 where type = 'bathymetry';
+----------+
| count(*) |
+----------+
| 59963    |
+----------+
1 row(s) fetched. 
Elapsed 0.009 seconds.

Object Store Profiling
> select table, path from list_files_cache();
+-------+---------------------------------+
| table | path                            |
+-------+---------------------------------+
| test  | release/2025-12-17.0/theme=base |
| test2 | release/2025-12-17.0/theme=base |
+-------+---------------------------------+
2 row(s) fetched. 
Elapsed 0.004 seconds.

Test cache associated with a table is dropped when table is dropped, and the other table with same path is unaffected.

> drop table test;
0 row(s) fetched. 
Elapsed 0.015 seconds.

Object Store Profiling
> select table, path from list_files_cache();
+-------+---------------------------------+
| table | path                            |
+-------+---------------------------------+
| test2 | release/2025-12-17.0/theme=base |
+-------+---------------------------------+
1 row(s) fetched. 
Elapsed 0.005 seconds.

Object Store Profiling
> select count(*) from list_files_cache() where table = 'test';
+----------+
| count(*) |
+----------+
| 0        |
+----------+
1 row(s) fetched. 
Elapsed 0.014 seconds.
> select count(*) from test2 where type = 'infrastructure';
+-----------+
| count(*)  |
+-----------+
| 142969564 |
+-----------+
1 row(s) fetched. 
Elapsed 0.013 seconds.

Object Store Profiling

Test that dropping a view does not remove cache

> create view test2_view as (select * from test2 where type = 'infrastructure');
0 row(s) fetched. 
Elapsed 0.103 seconds.

Object Store Profiling
> select count(*) from test2_view;
+-----------+
| count(*)  |
+-----------+
| 142969564 |
+-----------+
1 row(s) fetched. 
Elapsed 0.094 seconds.

Object Store Profiling
> drop view test2_view;
0 row(s) fetched. 
Elapsed 0.002 seconds.

Object Store Profiling
> select table, path from list_files_cache();
+-------+---------------------------------+
| table | path                            |
+-------+---------------------------------+
| test2 | release/2025-12-17.0/theme=base |
+-------+---------------------------------+
1 row(s) fetched. 
Elapsed 0.007 seconds.

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

alamb

Thank you @jizezhang -- this looks really nice. I think the basic idea of adding a table reference to the scope makes a lot of sense. I have a small suggestion for API design, but I think it should be straightforward

I will now go review #19388 and hopefully get that one through.

alamb · 2026-01-05T13:45:52Z

datafusion/execution/src/cache/list_files_cache.rs

 pub const DEFAULT_LIST_FILES_CACHE_TTL: Option<Duration> = None; // Infinite

+#[derive(PartialEq, Eq, Hash, Clone, Debug)]
+pub struct TableScopedPath(pub Option<TableReference>, pub Path);


I would personally suggest using named fields here so readers don't have to refer to the definition to know what .0 and .1 are -- so like

pub struct TableScopedPath { pub table: Option<TableReference>, pub path: Path }

I agree with this. I'm also wondering if it might make sense to allow user's who wish to override the default behavior to define their own scoping parameter by making this generic?

pub struct ScopedPath<S> { pub scope: Option<S>, pub path: Path, }

As long as it doesn't make the code too complicated, it seems like a good idea to me

Hi @BlakeOrth @alamb , regarding making a generic ScopedPath<S>, it seems to me that we would need to carry the generic parameter S to CacheAccessor<ScopedPath<S>, ...>, and to ListFilesCache<S> and then to CacheManager<S>? or there might be cleaner way to make it generic? Thanks!

This suggestion was mostly a "nice to have" if it ended up being very simple. I don' think it's strictly necessary and it sounds like it did not end up being simple! I don't think it's worth the additional complexity since the generic parameter would need to be propagated so far.

alamb · 2026-01-05T13:50:49Z

datafusion/core/src/execution/context/mod.rs

        {
            schema.deregister_table(&table)?;
+            if table_type == TableType::Base {
+                self.drop_list_files_cache_entries(&table_ref, &table_provider);


This API seems pretty specific and it isn't ideal in my mind that SessionContext now must check for a specific implementation -- I suggest a more generic method to the ListFilesCache trait so we can keep this logic in the implementation of ListingTable

https://github.com/apache/datafusion/blob/24e0983d4763c39cd5c6e7c414ae9d7629573736/datafusion/execution/src/cache/cache_manager.rs#L83-L82

Maybe something like

pub trait ListFilesCache { ... /// Notifies the cache that a specific table has been dropped. /// /// The table may clear up internal state that is specific to this table. pub fn table_dropped(table_ref: &TableReference) -> Result<()>; }

I moved the implementation to DefaultListFilesCache. One associated change is that given only TableReference we would need to loop through all entries in the cache to find those associated with the table, as opposed to getting the list of table paths directly from a ListingTable. I thought about maintaining an additional map from table ref to cache keys, but that seems to introduce a bit too much complexity and not sure if necessary for this use case. But please let me know if you have other suggestions. Thanks!

I personally feel like the looping strategy in the most recent commits is probably adequate until proven otherwise. Dropping tables should be occurring much less frequently than querying tables (at least in most normal usage), so in that sense it doesn't seem like it's in the "critical path" for performance. It seems to me that it's probably better to keep the implementation that is more simple and maintainable for the time being.

alamb · 2026-01-06T13:24:00Z

I merged #19388 -- I think we can finish this one up now

alamb · 2026-01-06T20:53:33Z

I rebased this PR against main

jizezhang · 2026-01-07T00:30:26Z

Thank you both for reviewing! I will work on addressing the comments and improving tonight:)

jizezhang · 2026-01-07T07:19:56Z

datafusion/core/src/datasource/listing_table_factory.rs

-                    table_path = table_path.with_glob(glob.as_ref())?;
+                    table_path = table_path
+                        .with_glob(glob.as_ref())?
+                        .with_table_ref(cmd.name.clone());


with_glob drops table_ref in the url, hence setting again

datafusion/datafusion/datasource/src/url.rs

Lines 321 to 326 in 1f654bb

pub fn with_glob(self, glob: &str) -> Result<Self> {

let glob =

Pattern::new(glob).map_err(|e| DataFusionError::External(Box::new(e)))?;

Self::try_new(self.url, Some(glob))

}

}

I think we could make table_path maintain with_glob. I did this change in

Alamb/suggestions jizezhang/datafusion#1

And it seems to have worked well

Thank you so much! I think I can merge the change, or would you like that in a separate PR?

xudong963 · 2026-01-07T09:31:53Z

datafusion/execution/src/cache/list_files_cache.rs

+        table_ref: &Option<TableReference>,
+    ) -> datafusion_common::Result<()> {
+        let mut state = self.state.lock().unwrap();
+        let mut table_paths = vec![];


Is it likely to be big? If so, it's better to use Vec::with_capacity()

My understanding is that the common case is one base path per table, but I could be wrong. Maybe we can wait and see what other reviewers think?

My understanding is the same as @jizezhang, I think the most common case will be a singe path per table, additionally Vec<Path> is effectively Vec<String> so I would never expect the actual memory allocated for the Vec itself to be very large.

That being said, it does look like we just end up needing a reference &path, so if we could avoid the path.clone() call that seems like it would be a nice improvement.

I am not sure how we would know how many entries we needed to remove here.

I think the reason they need to be copied out is that it isn't possible to modify the lru_queue while iterating over it as well.

If this becomes a performance problem, I think we could potentially solve it by actually splitting the cache somehow (eg. a map of maps) so we could drop all entries for a particular table atomically

I think the reason they need to be copied out is that it isn't possible to modify the lru_queue while iterating over it as well.

Yes that was exactly the reason.

alamb

Thank you so much @jizezhang and @BlakeOrth . I tested this out and it works great

I also went over the code and I think it looks good. I had a few suggestions on how to improve the code for your consideration, but nothing required

jizezhang#1

> create external table overturemaps
stored as parquet
location 's3://overturemaps-us-west-2/release/2025-12-17.0/theme=base/type=infrastructure';
0 row(s) fetched.
Elapsed 6.879 seconds.

> select table, path, metadata_size_bytes, expires_in, unnest(metadata_list)['file_size_bytes'] as file_size_bytes, unnest(metadata_list)['e_tag'] as e_tag from list_files_cache() limit 10;
+--------------+-----------------------------------------------------+---------------------+------------+-----------------+---------------------------------------+
| table        | path                                                | metadata_size_bytes | expires_in | file_size_bytes | e_tag                                 |
+--------------+-----------------------------------------------------+---------------------+------------+-----------------+---------------------------------------+
| overturemaps | release/2025-12-17.0/theme=base/type=infrastructure | 2750                | NULL       | 999055952       | "35fc8fbe8400960b54c66fbb408c48e8-60" |
| overturemaps | release/2025-12-17.0/theme=base/type=infrastructure | 2750                | NULL       | 975592768       | "8a16e10b722681cdc00242564b502965-59" |
| overturemaps | release/2025-12-17.0/theme=base/type=infrastructure | 2750                | NULL       | 1082925747      | "24cd13ddb5e0e438952d2499f5dabe06-65" |
| overturemaps | release/2025-12-17.0/theme=base/type=infrastructure | 2750                | NULL       | 1008425557      | "37663e31c7c64d4ef355882bcd47e361-61" |
| overturemaps | release/2025-12-17.0/theme=base/type=infrastructure | 2750                | NULL       | 1065561905      | "4e7c50d2d1b3c5ed7b82b4898f5ac332-64" |
| overturemaps | release/2025-12-17.0/theme=base/type=infrastructure | 2750                | NULL       | 1045655427      | "8fff7e6a72d375eba668727c55d4f103-63" |
| overturemaps | release/2025-12-17.0/theme=base/type=infrastructure | 2750                | NULL       | 1086822683      | "b67167d8022d778936c330a52a5f1922-65" |
| overturemaps | release/2025-12-17.0/theme=base/type=infrastructure | 2750                | NULL       | 1016732378      | "6d70857a0473ed9ed3fc6e149814168b-61" |
| overturemaps | release/2025-12-17.0/theme=base/type=infrastructure | 2750                | NULL       | 991363784       | "c9cafb42fcbb413f851691c895dd7c2b-60" |
| overturemaps | release/2025-12-17.0/theme=base/type=infrastructure | 2750                | NULL       | 1032469715      | "7540252d0d67158297a67038a3365e0f-62" |
+--------------+-----------------------------------------------------+---------------------+------------+-----------------+---------------------------------------+
10 row(s) fetched.
Elapsed 0.025 seconds.

> drop table overturemaps;
0 row(s) fetched.
Elapsed 0.003 seconds.

> select table, path, metadata_size_bytes, expires_in, unnest(metadata_list)['file_size_bytes'] as file_size_bytes, unnest(metadata_list)['e_tag'] as e_tag from list_files_cache() limit 10;
+-------+------+---------------------+------------+-----------------+-------+
| table | path | metadata_size_bytes | expires_in | file_size_bytes | e_tag |
+-------+------+---------------------+------------+-----------------+-------+
+-------+------+---------------------+------------+-----------------+-------+
0 row(s) fetched.
Elapsed 0.015 seconds.

> create external table overturemaps
stored as parquet
location 's3://overturemaps-us-west-2/release/2025-12-17.0/theme=base/type=infrastructure';
0 row(s) fetched.
Elapsed 1.273 seconds.

> select table, path, metadata_size_bytes, expires_in, unnest(metadata_list)['file_size_bytes'] as file_size_bytes, unnest(metadata_list)['e_tag'] as e_tag from list_files_cache() limit 10;
+--------------+-----------------------------------------------------+---------------------+------------+-----------------+---------------------------------------+
| table        | path                                                | metadata_size_bytes | expires_in | file_size_bytes | e_tag                                 |
+--------------+-----------------------------------------------------+---------------------+------------+-----------------+---------------------------------------+
| overturemaps | release/2025-12-17.0/theme=base/type=infrastructure | 2750                | NULL       | 999055952       | "35fc8fbe8400960b54c66fbb408c48e8-60" |
| overturemaps | release/2025-12-17.0/theme=base/type=infrastructure | 2750                | NULL       | 975592768       | "8a16e10b722681cdc00242564b502965-59" |
| overturemaps | release/2025-12-17.0/theme=base/type=infrastructure | 2750                | NULL       | 1082925747      | "24cd13ddb5e0e438952d2499f5dabe06-65" |
| overturemaps | release/2025-12-17.0/theme=base/type=infrastructure | 2750                | NULL       | 1008425557      | "37663e31c7c64d4ef355882bcd47e361-61" |
| overturemaps | release/2025-12-17.0/theme=base/type=infrastructure | 2750                | NULL       | 1065561905      | "4e7c50d2d1b3c5ed7b82b4898f5ac332-64" |
| overturemaps | release/2025-12-17.0/theme=base/type=infrastructure | 2750                | NULL       | 1045655427      | "8fff7e6a72d375eba668727c55d4f103-63" |
| overturemaps | release/2025-12-17.0/theme=base/type=infrastructure | 2750                | NULL       | 1086822683      | "b67167d8022d778936c330a52a5f1922-65" |
| overturemaps | release/2025-12-17.0/theme=base/type=infrastructure | 2750                | NULL       | 1016732378      | "6d70857a0473ed9ed3fc6e149814168b-61" |
| overturemaps | release/2025-12-17.0/theme=base/type=infrastructure | 2750                | NULL       | 991363784       | "c9cafb42fcbb413f851691c895dd7c2b-60" |
| overturemaps | release/2025-12-17.0/theme=base/type=infrastructure | 2750                | NULL       | 1032469715      | "7540252d0d67158297a67038a3365e0f-62" |
+--------------+-----------------------------------------------------+---------------------+------------+-----------------+---------------------------------------+
10 row(s) fetched.
Elapsed 0.018 seconds.

alamb · 2026-01-07T19:30:14Z

datafusion-cli/src/functions.rs

    }
 }

+// Implementation of the `list_files_cache` table function in datafusion-cli.


alamb · 2026-01-07T19:36:04Z

datafusion/execution/src/cache/list_files_cache.rs

+        table_ref: &Option<TableReference>,
+    ) -> datafusion_common::Result<()> {
+        let mut state = self.state.lock().unwrap();
+        let mut table_paths = vec![];


I am not sure how we would know how many entries we needed to remove here.

I think the reason they need to be copied out is that it isn't possible to modify the lru_queue while iterating over it as well.

If this becomes a performance problem, I think we could potentially solve it by actually splitting the cache somehow (eg. a map of maps) so we could drop all entries for a particular table atomically

alamb · 2026-01-07T20:11:41Z

datafusion/core/src/datasource/listing_table_factory.rs

-                    table_path = table_path.with_glob(glob.as_ref())?;
+                    table_path = table_path
+                        .with_glob(glob.as_ref())?
+                        .with_table_ref(cmd.name.clone());


I think we could make table_path maintain with_glob. I did this change in

Alamb/suggestions jizezhang/datafusion#1

And it seems to have worked well

alamb · 2026-01-08T14:34:07Z

In order to proceed with the branch-52 release, I am going to merge this and make a follow on PR to main with my suggestions. I'll then backport this code to branch-52

alamb · 2026-01-08T14:38:25Z

Follow on PR:

Improvements to list_files_cache table function #19703

## Which issue does this PR close?  - Builds on apache#19388 - Closes apache#19573 ## Rationale for this change  This PR explores one way to make `ListFilesCache` table scoped. A session level cache is still used, but the cache key is made a "table-scoped" path, for which a new struct ``` pub struct TableScopedPath(pub Option<TableReference>, pub Path); ``` is defined. `TableReference` comes from `CreateExternalTable` passed to `ListingTableFactory::create`. Additionally, when a table is dropped, all entries related to a table is dropped by modifying `SessionContext::find_and_deregister` method. Testing (change on adding `list_files_cache()` for cli is included for easier testing). - Testing cache reuse on a single table. ``` > \object_store_profiling summary ObjectStore Profile mode set to Summary > create external table test stored as parquet location 's3://overturemaps-us-west-2/release/2025-12-17.0/theme=base/'; 0 row(s) fetched. Elapsed 14.878 seconds. Object Store Profiling Instrumented Object Store: instrument_mode: Summary, inner: AmazonS3(overturemaps-us-west-2) Summaries: +-----------+----------+-----------+-----------+-------------+-------------+-------+ | Operation | Metric | min | max | avg | sum | count | +-----------+----------+-----------+-----------+-------------+-------------+-------+ | Get | duration | 0.030597s | 0.209235s | 0.082396s | 36.254189s | 440 | | Get | size | 204782 B | 857230 B | 497304.88 B | 218814144 B | 440 | | List | duration | 0.192037s | 0.192037s | 0.192037s | 0.192037s | 1 | | List | size | | | | | 1 | +-----------+----------+-----------+-----------+-------------+-------------+-------+ > select table, path, unnest(metadata_list) from list_files_cache() limit 1; +-------+---------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | table | path | UNNEST(list_files_cache().metadata_list) | +-------+---------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test | release/2025-12-17.0/theme=base | {file_path: release/2025-12-17.0/theme=base/type=bathymetry/part-00000-dd0f2f50-b436-4710-996f-f1b06181a3a1-c000.zstd.parquet, file_modified: 2025-12-17T22:32:50, file_size_bytes: 40280159, e_tag: "15090401f8f936c3f83bb498cb99a41d-3", version: NULL} | +-------+---------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 1 row(s) fetched. Elapsed 0.058 seconds. Object Store Profiling > select count(*) from test where type = 'infrastructure'; +-----------+ | count(*) | +-----------+ | 142969564 | +-----------+ 1 row(s) fetched. Elapsed 0.028 seconds. Object Store Profiling ``` - Test separate cache entries are created for two tables with same path ``` > create external table test2 stored as parquet location 's3://overturemaps-us-west-2/release/2025-12-17.0/theme=base/'; 0 row(s) fetched. Elapsed 14.798 seconds. Object Store Profiling Instrumented Object Store: instrument_mode: Summary, inner: AmazonS3(overturemaps-us-west-2) Summaries: +-----------+----------+-----------+-----------+-------------+-------------+-------+ | Operation | Metric | min | max | avg | sum | count | +-----------+----------+-----------+-----------+-------------+-------------+-------+ | Get | duration | 0.030238s | 0.350465s | 0.073670s | 32.414654s | 440 | | Get | size | 204782 B | 857230 B | 497304.88 B | 218814144 B | 440 | | List | duration | 0.133334s | 0.133334s | 0.133334s | 0.133334s | 1 | | List | size | | | | | 1 | +-----------+----------+-----------+-----------+-------------+-------------+-------+ > select count(*) from test2 where type = 'bathymetry'; +----------+ | count(*) | +----------+ | 59963 | +----------+ 1 row(s) fetched. Elapsed 0.009 seconds. Object Store Profiling > select table, path from list_files_cache(); +-------+---------------------------------+ | table | path | +-------+---------------------------------+ | test | release/2025-12-17.0/theme=base | | test2 | release/2025-12-17.0/theme=base | +-------+---------------------------------+ 2 row(s) fetched. Elapsed 0.004 seconds. ``` - Test cache associated with a table is dropped when table is dropped, and the other table with same path is unaffected. ``` > drop table test; 0 row(s) fetched. Elapsed 0.015 seconds. Object Store Profiling > select table, path from list_files_cache(); +-------+---------------------------------+ | table | path | +-------+---------------------------------+ | test2 | release/2025-12-17.0/theme=base | +-------+---------------------------------+ 1 row(s) fetched. Elapsed 0.005 seconds. Object Store Profiling > select count(*) from list_files_cache() where table = 'test'; +----------+ | count(*) | +----------+ | 0 | +----------+ 1 row(s) fetched. Elapsed 0.014 seconds. > select count(*) from test2 where type = 'infrastructure'; +-----------+ | count(*) | +-----------+ | 142969564 | +-----------+ 1 row(s) fetched. Elapsed 0.013 seconds. Object Store Profiling ``` - Test that dropping a view does not remove cache ``` > create view test2_view as (select * from test2 where type = 'infrastructure'); 0 row(s) fetched. Elapsed 0.103 seconds. Object Store Profiling > select count(*) from test2_view; +-----------+ | count(*) | +-----------+ | 142969564 | +-----------+ 1 row(s) fetched. Elapsed 0.094 seconds. Object Store Profiling > drop view test2_view; 0 row(s) fetched. Elapsed 0.002 seconds. Object Store Profiling > select table, path from list_files_cache(); +-------+---------------------------------+ | table | path | +-------+---------------------------------+ | test2 | release/2025-12-17.0/theme=base | +-------+---------------------------------+ 1 row(s) fetched. Elapsed 0.007 seconds. ``` ## What changes are included in this PR?  ## Are these changes tested?  ## Are there any user-facing changes?

alamb · 2026-01-08T14:51:25Z

I created the backport pR here:

[branch-52] Backport list_files_cache, and make default ListingFilesCache table scoped #19704

…sCache table scoped (#19704) ## Which issue does this PR close? - part of #18566 ## Rationale for this change Backport the fix for this regression into 52 release branch: - #19573 ## What changes are included in this PR? Backport these two commits to `branch-52` (cherry-pick was clean) - 1037f0a / #19388 - e6049de / #19616 <details><summary>Commands</summary> <p> ```shell andrewlamb@Andrews-MacBook-Pro-3:~/Software/datafusion$ git cherry-pick 1037f0a [branch-52 1fc70ac] feat: add list_files_cache table function for `datafusion-cli` (#19388) Author: jizezhang <[email protected]> Date: Tue Jan 6 05:23:39 2026 -0800 5 files changed, 446 insertions(+), 31 deletions(-) andrewlamb@Andrews-MacBook-Pro-3:~/Software/datafusion$ git cherry-pick e6049de Auto-merging datafusion/core/src/execution/context/mod.rs [branch-52 aa3d413] Make default ListingFilesCache table scoped (#19616) Author: jizezhang <[email protected]> Date: Thu Jan 8 06:34:10 2026 -0800 10 files changed, 474 insertions(+), 184 deletions(-) ``` </p> </details> ## Are these changes tested? By CI and new tests ## Are there any user-facing changes? A new datafusion-cli function and dropping a external table now clears the listing cache --------- Co-authored-by: jizezhang <[email protected]>

github-actions bot added documentation Improvements or additions to documentation core Core DataFusion crate catalog Related to the catalog crate execution Related to the execution crate datasource Changes to the datasource crate labels Jan 3, 2026

jizezhang force-pushed the table-scoped-lfc branch 3 times, most recently from 0bddd39 to 2376f35 Compare January 3, 2026 00:59

jizezhang mentioned this pull request Jan 3, 2026

Confusing behavior now required to to refresh the files of a listing table #19573

Closed

alamb reviewed Jan 5, 2026

View reviewed changes

alamb mentioned this pull request Jan 6, 2026

feat: add list_files_cache table function for datafusion-cli #19388

Merged

jizezhang added 2 commits January 6, 2026 15:53

make lfc table scoped

3e4f4e9

remove outdated test

c5ae6dc

alamb force-pushed the table-scoped-lfc branch from 2cd88bb to c5ae6dc Compare January 6, 2026 20:53

github-actions bot removed the documentation Improvements or additions to documentation label Jan 6, 2026

move code for dropping cache entries and update

ba65683

github-actions bot added the documentation Improvements or additions to documentation label Jan 7, 2026

update doc

8f20060

jizezhang commented Jan 7, 2026

View reviewed changes

jizezhang marked this pull request as ready for review January 7, 2026 07:30

xudong963 reviewed Jan 7, 2026

View reviewed changes

update doc

6c2cc9e

alamb added the api change Changes the API exposed to users of the crate label Jan 7, 2026

alamb changed the title ~~table scoped lfc~~ Make default ListingFilesCache table scoped Jan 7, 2026

alamb mentioned this pull request Jan 7, 2026

Alamb/suggestions jizezhang/datafusion#1

Closed

alamb approved these changes Jan 7, 2026

View reviewed changes

alamb mentioned this pull request Jan 7, 2026

Release DataFusion 52.0.0 (Dec 2025 / Jan 2026) #18566

Open

41 tasks

xudong963 mentioned this pull request Jan 8, 2026

[branch-52] Prepare 52.0.0 release version number and changelog #19661

Merged

alamb added this pull request to the merge queue Jan 8, 2026

Merged via the queue into apache:main with commit e6049de Jan 8, 2026
30 checks passed

alamb mentioned this pull request Jan 8, 2026

Improvements to list_files_cache table function #19703

Open

alamb mentioned this pull request Jan 8, 2026

[branch-52] Backport list_files_cache, and make default ListingFilesCache table scoped #19704

Merged

adriangb mentioned this pull request Jan 8, 2026

Refactor cache APIs to support ordering information #19597

Merged

	pub fn with_glob(self, glob: &str) -> Result<Self> {
	let glob =
	Pattern::new(glob).map_err(\|e\| DataFusionError::External(Box::new(e)))?;
	Self::try_new(self.url, Some(glob))
	}
	}

Make default ListingFilesCache table scoped #19616

Make default ListingFilesCache table scoped #19616

Uh oh!

Conversation

jizezhang commented Jan 3, 2026 • edited by alamb Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb commented Jan 6, 2026

Uh oh!

alamb commented Jan 6, 2026

Uh oh!

jizezhang commented Jan 7, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alamb commented Jan 8, 2026

Uh oh!

Uh oh!

alamb commented Jan 8, 2026

Uh oh!

alamb commented Jan 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jizezhang commented Jan 3, 2026 •

edited by alamb

Loading